C/C++ Dependency Generator

A python program that will scan a C/C++ program's source files and build a dependency file for each one.
Each dependency file will be created acording to a template and will have the list of dependent files for the corresponding source file, based on that file's #include directives.

The dependency files generated can then be included in the Makefile.

With this program it will be easy to keep your Makefile clean and efficient, recompiling the source files only when necessary.

Instructions

Setup

=> Running the program and Batch file:

In order to run the program the file main.py must be called from the command line or terminal.

The working directory from where the program is called will be considered the project's root directory, which will be used to search for files and to build absolute paths from relative ones.

To facilitate calling the program this repository comes with a batch file.

In the batch file the line python path\to\main.py must be adjusted by changing path\to\main.py with the path to the main.py file.

=> Configuring/Customizing the Program

• Dependency Template:

The dependency files will be generated based on a template which must be stored in a text file named dependency_template.txt, located somewhere in the project's directory tree.

This file can contain the following keywords, which will be replaced by the respective content, resulting in each source file's dependency file.

Keyword	Replace Value
|!dependents!|	The source file's list of dependent files, separated by whitespaces.
|!src_file_basename!|	The source file's basename (ex: main.cpp)
|!src_file_name!|	The source file's name (ex: main)
|!src_file_ext!|	The source file's extension (ex: cpp)

NOTE: If the configuration option include_source is set to True, the source file will be included in the |!dependents!| keyword's replace value always in the first position, allowing it's reference in the Makefile's rule via the use of the automatic variable $<.

For example, the following dependency_template.txt file

$(OBJ_DIR)/|!src_file_name!|.o: $(SRC_DIR)/|!src_file_basename!| |!dependents!| | $(OBJ_DIR)
	$(CC) $(LFLAGS) $< -o $@

can generate the following dependency file for a source file named Application.cpp

$(OBJ_DIR)/Application.o: $(SRC_DIR)/Application.cpp W:/productivity_monitor/headers/Application.h W:/productivity_monitor/headers/Cli.h | $(OBJ_DIR)
	$(CC) $(LFLAGS) $< -o $@

• Including the Dependency Files in a MakeFile:

The dependency files generated by this program will have the extension .d and receive the name of their corresponding source file.

For example, for a source file Application.cpp a dependency file Application.d will be generated.

The location where the dependency files will be stored depends on the value of the dependency_dir configuration option.

An example of a Makefile command that will include all dependency files is
-include $(SOURCES:$(SRC_DIR)/%.cpp=$(DEP_DIR)/%.d)

Where:

SRC_DIR stores the path to the directory where the source files are stored

DEP_DIR stores the path to the directory where the dependency files are stored

SOURCES stores all the source file names and can be built using, for example, SOURCES := $(wildcard $(SRC_DIR)/*.cpp)

• Configuration Options:

There are several configuration options that allow control over how the program will build the dependency files.
These configurations options can be changed directly in the program's command line interface, as detailed in the next section of this file.

The program has a set of default values for the configuration options, but a specific set can be created on a project by project basis.

A project's specific configuration set must be stored in a JSON file named dependency_config.json, located somewhere in the project's directory tree, which can be created directly from the program's command line interface (described in the "Using the Dependency Generator" section of this file).

NOTE: Until otherwise stated, all relative paths are assumed to be relative to the project's root directory (the current working directory from where the program was executed)

The available configuration options are:

Configuration	Data Type	Default Value	Description	Extra Details
sleep_timer	Integer	5	Number of seconds of wait between each source file scan cycle	Minimum = 1
dependency_dir	String		The path, absolute or relative, to the directory where all the dependency files and the project's configuration file will be stored	An empty string will cause the dependency files to be stored in the same directory as their respective source files and the project's configuration file in the project's root directory If this value changes after some dependency files have been generated, the program will move them to the new location
dependency_paths	Boolean	True	If True, the list of dependent files will have their absolute paths If False, only the basenames will be used
include_source	Boolean	True	If True, the source file will be added to the list of dependent files If False, it will not be included	The source file will always be in the first position
builtin_libs	Boolean	False	If True, language built in libraries will also be included in the dependent list If False, only custom libraries will be included	The program assumes that custom libraries are included using "" and built in libraries using <>
search_paths	String		The absolute paths, separated by ;, where files will be searched	For a more detailed explanation of the priority list of paths where files will be searched, consult the section "Technical Information" of this file
use_incomplete_list	Boolean	False	If True, the dependency file will be generated even if some of the dependent files couldn't be found If False, only if all dependent files are found will the dependency file be generated

NOTE: Including language built-in libraries as dependent files, by having the builtin_libs configuration set to True, will significantly increase the number of dependent files for each source file.

This will also increase the amount of time required by this program to crawl all necessary files and build their dependent lists.

However only the first couple of dependent lists built, for each execution of the program, will take longer to complete, with any subsequent lists being built in a short period of time.

For more information about what this program does to mitigate a project with large amount of dependent files, consult the section "Technical Information" of this file.

NOTE: In the cases where the builtin_libs configuration option is set to True and your operating system's PATH environmental variable doesn't contain the path to the mingw folder, it must be provided in the search_paths configuration option, otherwise the program will not be able to find the language built-in library files.

Using the Dependency Generator

This program has a command line interpreter that allows all necessary actions to be conducted directly from inside the program.

The current working directory from where this program is executed will be considered to be the project's root directory.

This is important since, in general, any relative paths provided to the program will be assumed to be relative to it.

=> Valid Commands

The valid commands to interact with this program are:

• run

Starts the scan of all the source files located in the project's directory tree and generates their respective dependency files as needed.

Only when any of the dependent files for a source file change will the dependent list be rebuilt and only if the dependent list changes will the dependency file be regenerated.

NOTE: To stop the scan process press CTRL-c

• config show

Shows a list of all the valid configuration options and their currently active values.

• config set key=value

Changes the currently active value for the configuration option key to the value of value.

In the cases where the desired value is "empty", for the configuration options of string data type, use ""

NOTE: This will not save the changes to the project's configuration file.

• config save

Saves the currently active configuration values into the project's configuration file, replacing any existing file.

NOTE: the file will be stored according to the value of the dependency_dir configuration option.

• config load

Loads the configuration values stored in the project's configuration file.

If the current project doesn't have a configuration file, the program's default configuration values will be loaded.

• config default

Loads the program's default configuration values.

• help

Displays a list of the valid commands and a brief description of each one.

• exit

Terminates the program.

Technical Information

Searching for Files

This program will crawl each source file searching for any #include directives. Their contents will then be converted into absolute paths to the included files.

In the cases where a valid absolute path can't be directly created from the contents of a #include directive, the program will search for a file with the same basename according to the following priority:

1. Project's root directory
2. The paths provided in the search_paths configuration option, in the same order
3. [if builtin_libs configuration is True] Any paths pointing to a mingw folder present in the PATH environmental variable of the operating system, in the same order

The first file found with the relevant basename will be the one chosen.

Optimizing for Large Amounts of Files

The most resource intensive task of this program is building each source file's dependent list, which is the list of absolute paths for all the files included by that source file.

This process involves crawling the source file and search for all #include directives. Then crawl those files and search for their included files. This process will continue until no more files in the include tree are left to crawl.

In this task, the two main resource consuming operations are:

1. Searching for files which a valid absolute path couldn't be directly built
2. Crawling each file

In order to reduce the amount of time required to build each source file's dependent list, the program stores the absolute paths of any relevant file it encounters while crawling files as well as the absolute path of all files included in each crawled file.

In practice this means that only the first couple to dependent lists built will require longer to complete. Any subsequent lists shouldn't require many files to be searched for since the program already found them while building previous lists.

This is more relevant the larger the project and especialy useful when including the built-in libraries.

The goal of this approach is to have the program crawl files at the start and then only when they are modified, as well as, only searching for a file in the hard drive once.

NOTE: The storage of this information is not permanent. When the program is terminated the collected information is lost and will have to be reaquired on the next execution.

However, it persists if the scan process is stopped and restarted later without terminating the program.

Including Files Before They Are Created

In the event that a file is included before it is created, it might be necessary to save the file with the #include directive after the included file is created, even if its contents haven't changed.

Consider the following example:

In file a.cpp, being tracked by this program, a new #include is added pointing to file b.cpp which hasn't been created yet and this change is saved.

Before file b.cpp can be created, the program identifies that file a.cpp was modified, crawls it again and correctly identifies that one of it's dependents couldn't be found.

The program didn't actualy add file b.cpp as a dependent of a.cpp, since it couldn't be found, and because the program stores the results of crawling a file it will not search for file b.cpp on it's own.

This means that even though the file has been created, the program will not crawl it, because it isn't being tracked.

To solve this issue, simply save file a.cpp again. This will change it's modify time triggering it to be crawled again and file b.cpp to be searched for again. This time the program should find it and start tracking it for any modifications.