Entity | A part of an organization that can be assessed as a single figure. |
Subsidiary | Same as an Entity on BitSight’s side. |
Group Cluster | A complex. It can contain entities/subsidiaries, or Groups, or more Group Clusters. |
Group | A structure that can contain Entities. |
In this blog post you will be introduced to the BitSight Automation Tool (https://github.com/NVISOsecurity/BitSight-Automation-Tool). BitSight Automation was developed to automate certain manual procedures and extract information such as ratings, assets, findings, etc. Automating most of these tasks is crucial for simplicity and time saving. Besides that, this tool also provides the possibility to collaborate with Scheduled Tasks and cronjobs. You can configure the tool to execute in certain intervals or dates, and retrieve the results from the desired folder without needing to interact with it.
What is BitSight? BitSight is a solution that helps organizations perform three (3) main functions.
It does all that by managing the company’s external facing infrastructure both automatically and manually, by allowing a company to provide updates to BitSight in order to keep their database up to date.
Other functions that are useful and provided by BitSight are:
By utilizing parts of the BitSight API Python wrapper developed by InfosecSapper, we developed an open source tool for the community to use, that fully automates some of BitSight’s operations, which we have named BitSight Automation Tool. This tool has a lot of potential to expand further with even more operations based on the needs that might arise.
You might be wondering by this point, what operations can this tool automate? Currently we have 5 operations that can be automated + 2 supplementary to assist with the tool’s maintenance.
The below image is a representation of the current state of the tool. At the time of writing the tool comes with the following structure.
In order to use the tool, we first need to install it. Regardless of the Operating System you are using, you will need to have Python installed. The tool has been tested with Python 3.8 and 3.11 at the time of writing, so any Python 3.x.x version should work.
Note: When installing Python make sure to include it in your PATH and include the PIP package manager as well.
Next step would be to install the tool’s requirements. To do so, navigate to the tool’s directory within a command prompt / terminal or PowerShell window and execute the following command: `pip install -r requirements.txt`
All the prerequisites are installed at this point, but we still have a couple more steps to perform before we can use the tool.
Now that you have installed the prerequisites, we are one step closer to utilizing the tool. Before we do so, we need to update a couple of files.
First you need to generate an API key from your account in BitSight. To do so,
In order for the BitSight Automation Tool to use this API key, you need to include it as an environmental variable in the system you will be running the tool on. We’ll do so below for both Windows and Linux.
For Windows systems,
For Linux systems,
echo export BITSIGHT_API_KEY={token} >> ~/.bashrc
This file is the heart of the tool. It will be queried to retrieve entities for every operation.
An example of a group_mapper.json file can be found below.
{
"Root":[
{"Group1": "Single Entity"},
{"Cluster Group2": ["EntityOne", "EntityTwo", "EntityThree"]},
{"Bigger Cluster Group3": [
{"SubCluster": ["Entity1", "Entity2"]},
{"SubCluster2": ["EntityUno", "EntityDos"]}
]},
"Random Entity that sits alone under the root"
]
}
The grouping subsidiaries can be organized like above. A few rules apply:
This file is the tie between BitSight and the BitSight Automation Tool. This is where the magic happens where your naming conventions can relate back to specific subsidiaries within BitSight.
An example of a guid_mapper.json file can be found below.
{
"Root": "463862495-ab29-32829-325829304823",
"Group1": "463862495-ab29-32829-325829304824"
}
This structure is the most basic structure you can have. The only thing you have to do is to create a new line for each subsidiary and assign its GUID from BitSight.
Below you may find how to get the GUIDs for the subsidiaries. A few rules that apply:
This step is mandatory for the tool to operate correctly. You can either use BitSight’s structure or you can create your own that suits best for your company.
Once you have completed the above steps, you need to modify one last item in the configuration of the tool.
The ‘groups.conf’ file structure should look like below (Figure 3)
You can add your groups one per line.
Note: Do not modify the first line. It should remain as is [Groups].
You have completed the manual part of the configuration! Pretty simple, right?
Execute the tool with the update operation.
python bitsight_automation.py update
This will go through Bitsight and find any subsidiaries that are missing. It will then prompt you to include it into the configuration. Follow the steps, provide the required information and the tool will take care of the rest.
Example:
PS C:\Users\Konstantinos Pap\Desktop\BitSight Automation> python .\bitsight_automation.py update
Subsidiary Name – GUID not found in our configuration
Would you like to include it (Y/N)? y
What is the name of this entity? Test
Under which group should this entity fall under({your-groups}) ? myTestGroup
Adding Subsidiary Name with guid {GUID} as Test in myTestGroup
Configuration Updated
Before we dive into how to utilize this tool, let’s first dive into how we can make an executable bundle for this tool. Performing this step allows for easier sharing and makes the tool usable by anyone, from analyst to CISO, without any specific requirements. We can use Pyinstaller to create the .exe standalone file.
Wait for a few seconds and notice there is a new directory created named “dist”. Grab the 2 .json files and the README.md file and copy them into the dist directory. Zip the contents of that directory and distribute that bundle on any Windows system. It will execute without any need for dependencies.
Note: You still have to export the environment variables to the new machines in order for the tool to be able to connect to BitSight.
Now that we installed and fully configured BitSight, we can go ahead and use its capabilities. As we already mentioned before, the tool allows for 5 different operations + 2 supplementary to assist with the tool’s maintenance.
We’ll first have a look at the usage menu of the tool and then we’ll navigate over a breakdown of each operation and how it works with examples.
Invoke the tool with its –help attribute.
PS ~/> python .\bitsight_automation.py --help
usage: bitsight_automation.py [-h] [-g {{your-groups}}] [-e ENTITY] [-v]
[-s {All,Critical-High,Critical,High,Low,Medium}]
[-so {alphanumerically,alphabetically}] [--search SEARCH] [--months MONTHS]
{rating,historical,findings,assets,reverse_lookup,list,update}
BitSight Automation tool to automate certain operations like historical report generation, findings categorization, asset list retrieval, reverse lookup of IP addresses and current ratings for entites
positional arguments:
{rating,historical,findings,assets,reverse_lookup,list,update}
The operation to perform.
optional arguments:
-h, --help show this help message and exit
-g {{your-groups}}, --group {{your-groups}} The group of entities you want to query data for.
-e ENTITY, --entity ENTITY A specific entity you want to query data for
-v, --verbose Increase output verbosity
-s {All,Critical-High,Critical,High,Low,Medium}, --severity {All,Critical-High,Critical,High,Low,Medium}
Level of Severity to be captured
-so {alphanumerically,alphabetically}, --sort {alphanumerically,alphabetically}
Sort rating results either alphanumerically or alphabetically.
--search SEARCH IP or Domain to reverse lookup for.
--months MONTHS Add in how many months back you want to view data for. If you want 1 year, fill in 12 months.
Max is 12
For any questions or feedback feel free to reach out to [email protected]
Now we will go through a breakdown of all different use cases within BitSight Automation Tool. We’ll go through the functional first and we’ll leave the last 2 complementary at the end.
For every operation different arguments will be required or not needed. The tool will let you know if you missed something during runtime. Example output:
[-] You need to specify one of the arguments --country or --region.
Use the rating operation to retrieve the current score of an entity or group in order to confirm if it’s above or equal to your policies. If a group is supplied this operation will output all of the subsidiaries under the specified group in the order you specified them in the JSON files (You also have the option to sort them alphanumerically)
Let’s try to fetch the current rating for our “Test” subsidiary.
PS ~/> python .\bitsight_automation.py rating -e Test
Test - 790
[+] Data saved to: 2023-03-17_bitsight_rating_Test.txt
Our Test Subsidiary has a score of 790. That’s an advanced score, so we can cross-verify with the company’s policies and take further action if needed. The results are also saved as a TXT file to allow easy copy/paste if required.
We can do the same thing for our Group and retrieve all the scores from all subsidiaries under our “Test Group”.
PS ~/> python .\bitsight_automation.py rating -g “Test Group”
[*] This may take a moment. Grab a coffee
Working on Test Group...
Test Group – 660
EntityOne - 620
Test Entity 2 - 760
EntityTwo - 770
[+] Data saved to: 2023-03-17_bitsight_rating_Test Group.txt
Notice we have retrieved ratings for all subsidiaries under “Test Group” in addition with the rating of “Test Group”. Some additional notes:
Use the historical operation to set up an overview of ratings for a given subsidiary or group over a specified timeframe (maximum 12 months) to showcase in reports and review progress or regress. Typically this operation is used with the -g argument but you can also utilize the -e argument for a given subsidiary only.
Let’s try to generate a report for our previous “Test Group” and its subsidiaries for the past year.
PS ~/> python .\bitsight_automation.py historical -g “Test Group” --months 12
Grab a coffee, this will take a while...
Working on Test Group...
[+] Data saved to 2023-03-17_Test Group_bitsight_historical_ratings_12_months.xlsx
Note: This command might take some time depending on the size of your organization + the number of subsidiaries it has to query data for. In any case, it is verbose enough to let you know in which group it is working on each time, so if you supplied a big cluster group you would have real time output of the progress.
The report:
There is a legend in the second sheet (tab) of the Excel file that denotes what these colors are and their scores – aligned with BitSight’s ratings and color coding.
Note: You can generate these types of reports with no limitation to a number of subsidiaries. You can even generate it for the entire organization using the Root subsidiary.
Use the findings operation to generate a filtered list of vulnerabilities for a subsidiary to remediate. This operation works solely with subsidiaries and not groups! You also need to supply the severity level with the -s argument.
Note: Your subsidiaries need to have a ‘Total Risk Monitoring’ subscription for this command to work. Otherwise it will produce an error.
Let’s retrieve the findings for our ‘EntityOne’ subsidiary under ‘Test Group’ we used earlier. We will retrieve the Critical vulnerabilities only.
PS ~/> Python .\bitsight_automation.py findings -e EntityOne -s Critical
[+] Data saved to bitsight_Critical_findings_EntityOne_2023-03-17.csv
Critical findings were downloaded and saved to a file called ‘bitsight_Critical_findings_EntityOne_2023-03-17.csv’. You can now start working on remediating the findings or assign it to the proper internal team.
Use the assets operation to retrieve the asset count and asset list of a subsidiary in order to validate your public IP space. This operation works solely with subsidiaries and not groups. This is a two-step process of querying. The operation first queries BitSight to retrieve the total count of public IPs in your subsidiary and then queries for the detailed asset list.
Note: This command requires a ‘Total Risk Monitoring’ subscription. If one is not available this command will produce an error.
Let’s attempt to retrieve the asset list for our ‘EntityOne’ subsidiary from the previous examples.
PS ~/ > python .\bitsight_automation.py assets -e EntityOne
EntityOne - 1410
*********** Asset List ************
[+] Asset List saved to: bitsight_asset_list_EntityOne_2023-03-17.csv
Note: This command will only fetch assets that are correctly attributed to this subsidiary. There’s a difference between correctly attributed by BitSight and internal/private Tagging.
Use this command to investigate where an IP, IP Range, domain or domain wildcard is attributed to and what IPs or domains it is associated with. This command only requires the –search argument.
Let’s attempt to find out where our test.com domain is attributed to and what public IPs it is associated with.
PS ~/> python .\bitsight_automation.py reverse_lookup --search test.com
test.com - ['<Redacted XX.XXX.XX.XXX>']: Found in: EntityOne
Use this operation to review the correlation between an entity’s custom given name and BitSight’s given name in a list for all defined entities. This command does not require any arguments.
Let’s view our subsidiaries and their correlation to BitSight.
PS ~/> Python .\bitsight_automation.py list
Listing Configuration...
Root – My Test BitSight Organization
Group One – First Group Subsidiary
EntityOne- Entity1 Test
Test Entity 2- Entity 2 Test
EntityTwo – SSEntity 2
Note: The mapping is {my JSON representation – BitSight’s representation}. The two names are bound over the GUID unique value for a subsidiary.
Use this operation to automatically update the tool and its respective JSON files. We already saw how this command works in the configuration section.
As we already mentioned, we can either manually execute the BitSight Automation Tool or we can set it up to automatically execute on its own recurringly over a specified window of time. This is relatively easy to achieve in both Linux and Windows operating systems.
To achieve this in Windows we need to utilize the Task Scheduler utility provided by Microsoft itself. No need to download or install any additional software. Let’s configure it.
Your Scheduled task is ready. You can manually invoke it once to verify it’s working correctly from the right bar, selecting ‘Run’
Note: You can follow this procedure for other tasks as well. (Update excluded as it requires manual intervention. However, the shell or prompt that will open will be interactive, so you can issue update comments on a daily basis anyway and if there are any, you can interact with the tool.)
The same process can be setup for Linux as well using the Cron Jobs it offers.
Write the following new line into the “/etc/crontab” file and replace ‘{your-tool-directory}’ with your tool’s directory (i.e. /opt/bitsight):
10 9 1 * * kali cd {your-tool-directory} && Python bitsight_automation.py historical -g Root –months 12
This will execute the tool every first of the month at 9:10 in the morning.
While executing this tool you might run into some issues here and there. This section will go over the 2 most common notifications you might encounter while using BitSight Automation.
You may have noticed in the Execution section of a couple of operations a note saying “This operation requires a ‘Total Risk Monitoring’ subscription to work. Otherwise it will produce an error”. These types of errors are usually encountered in Findings and Assets operations. The output will look something like this.
If we remove the ‘Total Risk Monitoring’ subscription from EntityOne and execute the findings operation on it again, we will run into the following error:
PS ~/> Python .\bitsight_automation.py findings -c EntityOne -s Critical
It appears as there are no findings in EntityOne or there is something wrong with the API. Please validate the old fashioned way using your browser.
More Details: list index out of range
It might be the case you do not have a 'Total Risk Monitoring' subscription. The 'Risk Monitoring' subscription is unable to work with the API for this operation
Response: {'links': {'next': None, 'previous': None}, 'count': 0, 'results': []}
In case you execute the tool and it reports back with “File not found” it means that somehow the necessary files were deleted. In order to resolve this issue you need to create the files again with the text “{}”inside them.
PS ~/> Python .\bitsight_automation.py findings -c EntityOne -s Critical
File not found: 'group_mapper.json'. Please copy the 'group_mapper.json' to the same directory as this tool and try again.
This blogpost presented the BitSight Automation Tool as a valuable enhancement for organizations employing BitSight for performing external assessment and reducing exposure, as their solution.
Some key perks of this tool are as follow:
Konstantinos is a Senior Cybersecurity Consultant at NVISO Security.
With a background in software engineering, he has an extensive set of skills in coding which helps him in day-to-day operations even in the Cybersecurity area. His motto; “Better spend 5 hours debugging your automation, than 5 minutes performing an automatable task”.