
SAFE API Connector (for CTEM)
Outline
Objective
This guide shows you how to send asset and security-finding data from your own systems into SAFE using the API Connector, so that SAFE can use it for exposure prioritization.
Typical sources are:
- A security data lake such as Databricks, Snowflake or Redshift
- A CMDB or asset inventory you maintain
- Exports from security tools, such as SentinelOne, Tenable or ServiceNow
By the end of this guide you will have:
- Created API credentials in SAFE and obtained an access token.
- Exported your data as compressed NDJSON files.
- Requested secure upload URLs and uploaded your files to SAFE.
- Confirmed that the data appears on the Assets and Findings pages, labelled with your source name.
Target Users
This document is intended for Engineers and Architects who wish to understand how to post security findings to SAFE. You do not need prior SAFE API experience, but you should be comfortable running curl commands or a short script.
- SAFE Administrator: Creates the API credentials in SAFE and confirms the data after upload.
- Security or Data Engineer: Ingests data from the data lake, builds the NDJSON files, and automates the push.
Overview
The API Connector lets you send asset and security-finding data to SAFE from any system you control: a security data lake (for example Databricks, Snowflake or Redshift), a CMDB, or exports from your security tools, such as SentinelOne, Tenable or ServiceNow.
You export your data as compressed NDJSON files, ask SAFE for upload URLs, and upload the files over HTTPS. SAFE then matches each asset to the assets it already knows, merges it, and shows the data on the Assets and Findings pages, labelled with the source name you sent.
Key terms
- Asset: a device SAFE tracks, such as a server, laptop, network device or cloud resource.
- Signal: one line in your file. It describes one asset and everything one source knows about it: identity, hardware, cloud details, software, patches, findings, compliance results and compensatory controls.
- Source: the tool the data came from, such as Qualys or Tenable. SAFE shows it as the asset's and finding's source. A file can carry one source or several.
- Finding: a security issue on an asset: a vulnerability (
va) or a configuration control result (ca).- Push: one batch of files you upload. Each push is the full, current picture of every asset it contains, for each source that reports the asset.
- Upload URL: a short-lived, pre-signed HTTPS link from SAFE that accepts exactly one file, of the size and SHA-256 you declared.
How it works
| Step | What you do | What SAFE does |
|---|---|---|
| 1. Authenticate | Call POST /api/v3/auth with your API credentials. |
Returns an access token, valid for 60 minutes. |
| 2. Prepare files | Write your asset records as NDJSON, gzip each file, then note each file's size in bytes and its SHA-256. | — |
| 3. Request upload URLs | Call POST /api/v3/integrations/file-ingest/uploads with the size and SHA-256 of each file. |
Returns one secure, time-limited upload URL per file. |
| 4. Upload | Upload each file to its URL with an HTTP PUT. | Stores the file in your SAFE storage. A 200 OK confirms SAFE has it. |
| 5. Review | Nothing more to do. | Processes the files in the background and shows the data on the Assets and Findings pages. |
Prerequisites
On the SAFE side
- SAFE Admin Access
- SAFE API base URL: The URL you use to sign in, for example https://{your-tenant}.safescore.ai
- IP address is allowed: If your tenant restricts access by IP, the machine that calls the SAFE APIs must be on the allowed list
On the Source side
- Read access to data
- A way to export to NDJSON: A query plus a script, notebook or ETL job that writes one JSON object per line. Any language that can write JSON and gzip files works.
- A place to run the push: A server, container or scheduled job with
curlor an HTTP client. It must reach your SAFE URL over HTTPS and the upload URLs SAFE returns, which are hosted on Amazon S3 - A secrets manager: A place to store the API username and password.
- A scheduler (recommended): Pushes should repeat, typically every 24 hours. Use your existing orchestrator or cron.
Data requirements
For every asset you send, make sure your data lake (source) can provide:
- A source name. The name of the tool the data came from, for example
Qualys. It is case-sensitive and appears in SAFE as the asset's source, so keep it identical on every push. - At least one identity field. One of
cloudResourceId,hostname,fqdn,netbiosName, a MAC address or an IPv4 address. SAFE uses these to match the record to assets it already knows. - A complete picture of the asset for that source. Each push replaces what that source previously reported for the asset: findings, software, patches and tags missing from the new record are marked as resolved. Plan to send the full current list every time, not only changes.
- Open vulnerabilities only. A vulnerability left out of the next push is closed for that source.
- Everything else (hardware, cloud details, owners, software, patches, compliance results) is optional. Send what you have; more context gives SAFE better prioritization.
Limitations
| Limit | Value |
|---|---|
| Files per upload URL request | 10 |
| Maximum file size (compressed) | 250 MiB (262,144,000 bytes). A larger sizeBytes is rejected with a 400 that states the limit. |
| Upload URL lifetime | 10 minutes; see expiresAt |
| Access token lifetime | 60 minutes |
| Upload URLs per tenant | Currently 10 upload URLs per minute, shared by every client of your tenant. URLs are counted, not requests: a request for 10 files uses 10. A request that would pass the limit is refused whole with 429 and uses none of it; wait the number of seconds in its Retry-After header. |
| File format | gzip only. A file that is not gzip (plain NDJSON included), or a corrupt gzip file, is rejected whole and none of it is imported. |
| Size of one line | At most 128 MiB per line (one signal), uncompressed. A longer line is skipped; the rest of the file is still processed. There is no limit on the number of lines per file beyond the file size. |
| Findings per asset | 20,000. An asset with more findings in one signal is not imported. |
Step-by-step Procedure With Examples
In the examples below, replace https://{your-tenant}.safescore.ai with the URL you use to sign in to SAFE. If you are not sure of your API base URL, ask your SAFE representative.
Step 1: Authenticate
Create API credentials
- In SAFE, go to Settings → API Credentials and select +.
- Enter a name (for example
data-lake-uploader), choose an expiry and select Add. - Copy the API username and password that SAFE shows, and store them in a secrets manager.
Get an access token
Exchange your API credentials for an access token with HTTP Basic authentication:
1 2 |
|
The response contains accessToken. Send it as Authorization: Bearer {accessToken} on the upload URL request in Step 3.
- A token is valid for 60 minutes. Reuse it for all calls in that time; do not authenticate before every call.
- If a SAFE API call returns
401, the token has expired or is invalid: authenticate again and retry the call once. - The file uploads in Step 4 go to S3 and do not use the token.
- SAFE does not accept an API password that contains a colon (
:).
Reference script: authentication and token renewal
A minimal Python 3 client (needs the requests package) that gets a token, reuses it for its 60-minute life, renews it 5 minutes before expiry, and on a 401 authenticates again and retries the call once. Save it as safe_auth.py; the example push script below uses it.
Show the script: safe_auth.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
|
Step 2: Prepare your files
File format
- NDJSON (newline-delimited JSON), UTF-8: one signal per line, written as
{"signals":[ {one signal} ]}. Each line is a complete JSON object; lines are separated by\n. Do not wrap the lines in an array, and do not put more than one signal in a line'ssignalsarray: such a line is skipped. - gzip-compressed. The upload URL is signed for
application/gzip, and files must be gzip: a file that is not gzip (plain NDJSON included) is rejected whole, whateverContent-Typeis sent. Always gzip the file before measuring and hashing it. Any local file name works;*.ndjson.gzis a useful convention. - Sources. A file can carry one source or several. If it carries several, keep each source's rows together (for example, sort the file by
entity.source); rows of more than a few sources mixed throughout a file are split into small batches and take longer to process. Keep each source's assets in as few, full files as possible: SAFE closes a source's last, partly filled batch when the file ends, so a source spread thinly over many files also creates many small batches. One file per tool is the simplest way to do both. - Put many signals in each file, within the limits in Limits. Fewer, larger files are better than many small ones: do not send one asset per file.
- Send each asset once per file. Do not split one asset across lines or files.
Sample record. One asset with every field SAFE accepts, so you can see the full structure. In your file the same record is written on one line; it is shown formatted here for reading. Send only the fields you have. compensatoryControls is shown here for completeness; in practice, send it in the security tool's own files (for example under source SentinelOne).
Show the sample record (formatted for reading; send it as one line)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 |
|
Record structure
Each line is one signal. version, id, type and entity.source are required, plus at least one identity field. Every other section is optional: send only the sections and fields you have.
| Section | What it holds | Required inside each item |
|---|---|---|
version, id, type |
The envelope: always "3.0.0", your UUID for the line, always "default" |
All three |
entity |
source, sourceAssetId, firstSeen, lastSeen and entityAttributes |
source |
entity.entityAttributes |
The device: names, network interfaces, type, tags, platform, hardware and OS, cloud details, location, owners | At least one identity field (see below) |
securityContexts[] |
Findings: vulnerabilities (type: "va") and configuration control results (type: "ca") |
type, name, status.complianceStatus; a ca also needs controlId and policyId or policyName |
softwares[] |
Installed software and agents (not the operating system) | name |
patches[] |
Patches (KB numbers) | patchNumber, status |
compensatoryControls[] |
Security-tool configuration evidence (EDR …), one block per type. Send it in the security tool's own files, for example under source SentinelOne |
type, and at least one control with controlId, controlName and value |
Every field, with its format and meaning, is listed in Field reference below.
Asset identity: how SAFE recognises a device
Every signal needs at least one identity field: cloudResourceId, hostname / fqdn / netbiosName, networkInterface[].macAddress or networkInterface[].ipv4. SAFE treats two records as the same asset at the first rule that matches:
- Cloud resource ID is the same (
cloudResourceId). Always send it for cloud resources. - Host name is the same (
hostname,fqdnornetbiosName; not case-sensitive;WEB01matchesWEB01.corp.example.com), and the serial numbers do not differ. - MAC address is the same, when one of the two records has no host name. Only the primary interface's IP and MAC (
isPrimary, else the first valid one) are used for matching; other interfaces are shown, not matched. - IP address is the same, when one of the two records has no host name. The weakest rule: IPs change and public IPs can be shared (NAT), so send private addresses where you can.
Never merged: different serial numbers, different cloud resource IDs, or different host names (host1.eu.example.com and host1.uk.example.com stay separate). A serial number cannot be an asset's only identity field; sent with an IP or MAC address, a matching serial can link two records.
Note
Host name and FQDN together. When
fqdncontains a dot, SAFE keepshostnameandnetbiosNameonly if they are the same asfqdn; otherwise it keeps the FQDN alone. Matching is not affected (a short name still matches its FQDN), but the short names are not shown. If you have an FQDN, send it infqdn; sendhostnamealone only when there is no FQDN.
Note
Skipped records. A record with no usable identity is skipped. Placeholders do not count:
0.0.0.0,127.0.0.1,169.254.x.x, IPv6 addresses,00:00:00:00:00:00, and serials such asN/A. A virtual-machine MAC address (VMware, VirtualBox, Docker and similar) is stored but never used to match. A generic host name such asLINUX,UBUNTUorIPHONEneeds a MAC address with it, or the whole record is skipped.
The full-snapshot rule
For each asset and source in a push, SAFE treats the record as the complete, current picture:
- Items in the new signal are added or updated.
- Items SAFE had from that source for that asset, but that are missing from the new signal, are marked as gone (resolved, not deleted). This applies to findings, software, patches, asset tags, location, and compensatory controls (when compensatory-control assessment is enabled for your tenant).
- Assets that are not in the push are not changed.
- Data from other sources on the same asset is not changed.
So always send the full list of findings, software, patches and compensatory controls for an asset, not only what changed since the last push. Send open vulnerabilities only: a vulnerability missing from the source's next signal for that asset is closed.
1 2 3 4 |
|
Source
entity.sourcenames the tool the data came from, for exampleQualys. It is required on every line. A file may contain lines from several sources; each line is processed under its own source.- Any name is accepted, exactly as you send it. Names are case-sensitive:
Qualysandqualysare two different sources. Keep them consistent between pushes. - A name may be up to 255 characters, must not be only digits, and must not be
manual. - If two tools report the same asset, send each tool's data under its own source name, in its own files or as its own group of rows. SAFE merges them into one asset.
Values
| Kind | Format |
|---|---|
| Dates and times | ISO 8601 with a time zone, for example 2026-09-30T18:00:00Z or 2026-09-30T23:30:00+05:30. Always include the zone: a time without one, or a date alone, is read as UTC (a date as midnight), which shifts a local time. |
| Numbers | Integers, for example "cpuCores": 40. A numeric string or a decimal is also accepted; decimals are truncated. |
| JSON types | Send each value in the type the field reference gives. A number or true/false in a text field is kept as text; an object or array in a single-value field drops only that field. Identity fields (hostname, fqdn, netbiosName, ipv4, macAddress, cloudResourceId), finding names, envelope fields, severity and timestamps must be JSON strings, or the field is dropped. A wrong structure, such as hardware or status sent as a string, skips the whole line. |
| Finding type | va (vulnerability) \ |
| Finding status | status.complianceStatus: fail \ |
| Finding severity | severity (a plain string, for example "severity": "high"): critical \ |
| Software type | application \ |
| Patch status | INSTALLED \ |
| Location | An array with one full country name, for example ["United States"] (a single string is also accepted). Use the same spelling every time. |
| Owners | One string per owner field; a list is accepted and joined with ", ". |
| Keys | camelCase, exactly as in the field reference. Unknown keys are ignored and the line is still processed. (Unknown fields in the upload URL request are rejected with 400.) |
Field reference
Every key is optional unless the Required column says otherwise. Keys not listed here are ignored.
Envelope and entity
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
version |
Yes | "3.0.0" exactly | Contract version of the record. Any other value skips the whole line. |
id |
Yes | UUID (any version) | Your own unique id for this line. Must be a valid UUID; a line with an invalid id is skipped. Not stored: SAFE assigns its own asset id. |
type |
Yes | "default" exactly | Record type. Any other value skips the line. |
entity.source |
Yes | String, 1–255 bytes (spaces trimmed); not digits only; not manual in any case. Case-sensitive. |
Name of the tool or system that reported the asset, for example Qualys. Shown as the asset's source. A finding this source reported before and is missing from its newest signal for the asset is closed. |
entity.sourceAssetId |
No | String | The asset's id inside the source tool. Kept by SAFE for traceability; not shown. |
entity.firstSeen |
No | ISO 8601 timestamp | When the source first saw the asset. Shown as the source's first-seen date; set when SAFE first records the asset for this source and not moved by later files. |
entity.lastSeen |
No | ISO 8601 timestamp | When the source last saw the asset. Shown as the source's last-seen date; updated on every push. |
Identity, network, location and owners (entity.entityAttributes)
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
hostname |
One of the identity fields | String (must be a JSON string); shown uppercased | Short host name. Identity key; shown in Network Interfaces. See Host name and FQDN together. A name that only wraps an IP (for example ip-10-1-2-3) is not used as a host name; SAFE uses the IP instead. |
fqdn |
One of the identity fields | String (must be a JSON string), for example web-prod-01.corp.example.com; shown uppercased | Fully qualified DNS name. Identity key; shown in Network Interfaces; used for the asset name when there is no cloud name. |
netbiosName |
One of the identity fields | String (must be a JSON string), up to 15 characters; shown uppercased | Windows NetBIOS name. Identity key; shown in Network Interfaces. |
networkInterface[].ipv4 |
One of the identity fields | Dotted IPv4 string. 0.0.0.0, 127.0.0.1 and 169.254.x.x are dropped; IPv6 is not accepted. Public addresses are accepted, but send private addresses where you can: a shared public IP can match unrelated devices. | IPv4 address of one interface. Identity key; shown in Network Interfaces (newest first). |
networkInterface[].macAddress |
One of the identity fields | String: aa:bb:cc:dd:ee:ff, aa-bb-…, aabb.ccdd.eeff or 12 hex digits; stored lowercase. 00:00:00:00:00:00 is dropped. | MAC address of the same interface. Identity key; shown in Network Interfaces. |
networkInterface[].isPrimary |
No | true \ | false ("yes"/"no" and 1/0 are also accepted). The first true wins; without one, the first valid interface is primary. |
sourceAssetType |
No | String | Asset type as the source names it, for example Server. Shown as sent; SAFE derives its own asset category separately. |
sourceAssetSubtype |
No | String | Asset subtype as the source names it, for example Web Server. Shown as sent. |
assetTags[].value |
No | Non-empty string; tags made only of special characters are dropped | A free-form tag. Shown as an asset tag; filterable. |
platform |
No | windows \ | linux \ |
location |
No | Array with one full country name, for example ["United States"] (a single string is also accepted). Only the first value is used; shown uppercased. Use the same spelling every time. | Country the asset is in. Shown as the asset's location; filterable. |
businessOwner, itOwner, vmOwner, deviceOwner |
No | One string each; a list is accepted and joined with ", " | Owners of the asset (name or email). Shown on the asset. When several sources report an owner, SAFE shows one source's value. |
Hardware and operating system (entity.entityAttributes.hardware)
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
osName, osVersion, osVendor |
No | String | Operating system name, version and publisher. Shown under Hardware and as the OS software row. SAFE normalises them (for example "Microsoft Windows Server 2019 Datacenter" may show as "Windows Server 2019"). |
osArch |
No | String, for example x64 | OS architecture. Shown under Hardware. |
osReleaseInstallDate |
No | ISO 8601 timestamp | When the current OS release was installed. Shown under Hardware. |
osInstallTime |
No | ISO 8601 timestamp | When the OS was first installed. Shown as the install date of the OS software row. |
osKernel |
No | String, for example 10.0.17763.5329 | Kernel or full build number. Not shown; SAFE uses it to work out the Windows build when osVersion has none, which drives patch state and CVE closure. |
serialNumber |
No | String; placeholders such as N/A or "To be filled by O.E.M." are ignored | Hardware serial number. Shown under Hardware. Two different serial numbers are never merged. A serial number cannot be an asset's only identity field; sent with an IP or MAC address, a matching serial can link two records. |
systemManufacturer, systemModel |
No | String | Hardware maker and model. Shown under Hardware; used to recognise virtual machines and device types. |
cpuModelName |
No | String | CPU model. Shown under Hardware. |
cpuCores, totalPhysicalMemMB |
No | Integer (a numeric string or a decimal is accepted; decimals are truncated) | Number of CPU cores and RAM in megabytes. Shown under Hardware. |
biosVendor, biosVersion |
No | String | BIOS vendor and version. Shown under Hardware. |
biosReleaseDate |
No | ISO 8601 timestamp | BIOS release date. Shown under Hardware. |
Send the operating system in hardware, not in softwares. Send agents as softwares.
Cloud (entity.entityAttributes; used only when cloudResourceId is present)
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
cloudResourceId |
One of the identity fields | String (must be a JSON string): AWS ARN, Azure resource ID or GCP URI, at most 512 characters; stored lowercase. A placeholder such as -, 0, n/a, none or null counts as no id. | The cloud's unique id for the resource. Strongest identity key; marks the asset as a cloud asset. Without a valid id, the other cloud fields are ignored. |
cloudServiceProvider |
No | AWS \ | Azure \ |
cloudRegion |
No | String, for example us-east-1 | Cloud region. Shown under Cloud. |
cloudAccountId |
No | String (a number is accepted) | Cloud account, subscription or project id. Shown under Cloud. |
cloudNativeAssetType |
No | String, for example AWS::EC2::Instance | The provider's own resource type. Shown under Cloud; drives cloud categorisation. |
cloudAssetType |
No | String, for example ec2-instances | Resource type in plain words. Shown under Cloud. |
cloudInstanceId |
No | String, for example i-0abc123 | Instance id. Shown under Cloud. |
cloudAssetName |
No | String | Resource name in the cloud. Shown under Cloud; becomes the asset's display name. |
cloudVpcId, cloudSubnetId, cloudResourceGroupName |
No | String | VPC, subnet and resource group. Shown under Cloud. |
containerInfo[].containerName |
Yes, per item | Non-empty string | Name of a container running on the resource. The number of distinct names is the container count. |
containerInfo[].containerCount |
No | Integer (a numeric string is accepted) | Running instances of that container. Shown with the image attributes. |
imageAttributes.registry, namespace, name, imageTags, digest |
No | String | The container image the resource runs. Shown under image attributes. |
Vulnerabilities (securityContexts[] with type: "va")
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
type |
Yes | "va" | Marks the finding as a vulnerability. |
name |
Yes | For a CVE, exactly one bare CVE id, for example CVE-2023-44487 (one finding per CVE). Otherwise the title. | The vulnerability. For a known CVE, SAFE adds the NVD description, CVSS and EPSS. |
status.complianceStatus |
Yes | fail \ | pass |
severity |
No | Plain string: critical \ |
high \ |
sourceFindingType, sourceFindingSubtype |
No | String | Your source's own category and sub-category. Shown as sent. |
description.technical |
No | Text | Technical description. Shown; for a known CVE the NVD text replaces it. |
openSinceInSource, lastAssessed |
No | ISO 8601 timestamp | When the source first found it and last checked it. Shown. |
remediation.description |
No | Text | How to fix it. Shown as remediation. |
threatTags |
No | Array of strings | Threat tags from the source. Used for CVEs. Recognised tags (exact text, case-sensitive): CISA Known Exploit (marks the CVE as CISA KEV), Known Exploit, Ransomware Linked, Malware Linked, Exploit Kit Available. Other tags are kept but do not affect scoring. |
affectedSoftware.name, version, vendorName, type, installPath, installDate |
name, when affectedSoftware is sent |
As in softwares[]; do not use an OS type. |
The software the vulnerability is in. Use the same values as the matching softwares[] entry so SAFE links the two; any difference in name, version or install path creates a second software row. |
Configuration controls (securityContexts[] with type: "ca")
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
type |
Yes | "ca" | Marks the finding as a configuration control result. |
name |
Yes | Text | The control's statement. Shown; replaced by the CIS title when SAFE recognises the control. |
controlId |
Yes | String, for example 18.9.47.4.1 | The control's id in its benchmark. Shown as the control id and used to look the control up in SAFE's CIS library. |
policyId, policyName |
At least one | String | The policy the control belongs to. Controls with the same policyId and policyName are grouped into one policy. For CIS policies SAFE generates the policy id from the name. |
complianceFramework |
No | String, for example CIS | Framework. Shown. |
status.complianceStatus |
Yes | fail \ | pass |
severity |
No | Plain string: critical \ |
high \ |
sourceFindingType, sourceFindingSubtype |
No | String | Your source's own category and sub-category. Shown as sent. |
description.technical, remediation.description |
No | Text | Why the control matters and how to pass it. Shown; the CIS text replaces them when SAFE recognises the control. |
openSinceInSource, lastAssessed |
No | ISO 8601 timestamp | When the source first and last evaluated the control. Shown. |
Software, patches and compensatory controls
| Field | Required | Format / allowed values | Meaning and what SAFE does |
|---|---|---|---|
softwares[].name |
Yes | Non-empty string | Product name. Shown in the software inventory; SAFE may normalise it. |
softwares[].vendorName, version, installPath, installDate |
No | String; installDate is an ISO 8601 timestamp | Vendor, version, install location and install time. Shown. Name, version and install path identify the software row. |
softwares[].type |
No | application \ |
cpe |
softwares[].kernel |
No | String | Kernel the software is built for. Not shown; improves SAFE's product matching. |
patches[].patchNumber |
Yes | String, for example KB5034439 (a number is kept as text) | Patch or KB id. Not listed in the UI; SAFE uses patches to compute software patch state and to close the CVEs a patch fixes. |
patches[].status |
Yes | INSTALLED \ | PENDING \ |
patches[].installDate |
Recommended for an INSTALLED patch | ISO 8601 timestamp | When it was installed. Without a date, an installed patch is kept but cannot close any CVE. |
compensatoryControls[].type |
Yes | String, for example EDR; one block per type | Kind of protection. |
compensatoryControls[].controls[].controlId, controlName, value |
Yes | Text; a number or true/false is kept as text | Each configured setting. Used to assess whether the control reduces the risk of a finding (SentinelOne EDR today). |
Example: one complete signal, source Qualys (formatted for reading; send it as one line)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 |
|
Sample files: Qualys, Tenable and SentinelOne (one source per file)
Three ready-to-send files, one source each. The server WEB-PROD-01 appears in all three; SAFE merges it into one asset that lists Qualys, Tenable and SentinelOne. Every line passes the file checks: no line or field is skipped. gzip each file before upload..
qualys-sample.ndjson — Qualys: an on-premises server with vulnerabilities, a CIS control, software and patches; a cloud instance with containers; a laptop.
1 2 3 |
|
tenable-sample.ndjson — Tenable: the same server, with a CPE software row and the vulnerability linked to it.
1 |
|
sentinelone-sample.ndjson — SentinelOne: the same server, with EDR compensatory controls.
1 |
|
Compress, then measure
Each upload URL is signed for one exact file size and SHA-256. So always:
- Write the NDJSON and gzip it to a file.
- Read the file's size in bytes from disk (for example
os.path.getsize()orstat), and compute its SHA-256 (for exampleopenssl dgst -sha256 -binary FILE | base64). - Request the upload URLs with those sizes and SHA-256 values (Step 3).
- Upload those same files (Step 4).
Warning
Do not re-compress, re-encode or otherwise change a file after measuring it. gzip output changes with the tool, level and timestamp, so a re-compressed file has a different size or content, and S3 refuses the upload (
403for a different size,400BadDigestfor different content).
Step 3: Request upload URLs
Ask for one upload URL per file, listing each file's size and SHA-256 in the order you will upload them:
1 2 3 4 |
|
To compute the sha256 of a file: openssl dgst -sha256 -binary assets-part-1.ndjson.gz | base64
| Rule | Limit |
|---|---|
| Body | Required. Exactly {"files": [{"sizeBytes": N, "sha256": "..."}, ...]}; any other field is rejected (400). |
| Files per call | 1 to 10 |
sizeBytes |
Whole number, the exact size in bytes of the gzipped file, from 1 up to the maximum file size |
sha256 |
Required. The standard base64 (44 characters, with padding) of the raw 32-byte SHA-256 of the exact gzipped file you will upload; not hex. For example `openssl dgst -sha256 -binary FILE \ |
| Upload URLs per tenant | Limited per minute; see Limits |
A successful call returns 201:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
|
uploadsis in the same order asfiles:uploads[0]is for the first file you listed.- Each upload has an
uploadId. Log it with your file name: SAFE support uses it to trace a file. Do not send it back. expiresAtis when the URLs stop working, currently 10 minutes after the call. Use this value rather than assuming a fixed duration.- Hash and measure the final
.gzfile. If it changes after you request its URL (for example, it is re-compressed), the upload is refused. - Request URLs only when the files are ready, and only for as many files as you can upload before
expiresAt.
Step 4: Upload the files
Upload each file with an HTTP PUT to its url, sending exactly the headers listed for it and the file as the raw request body:
1 2 3 4 5 |
|
curl sets Content-Length from the file; it must equal the sizeBytes you sent. $SHA256 is the sha256 you sent for this file.
| S3 response | Meaning | What to do |
|---|---|---|
200 OK |
SAFE has the file. | Done. Processing starts automatically. |
412 Precondition Failed |
A file was already uploaded to this URL. | Nothing; the first upload stands. A file cannot be overwritten. |
400 BadDigest |
The file's content differs from the sha256 you sent: it changed after you hashed it, or a different file was uploaded. |
Do not retry this file to this URL. Request a new URL with the file's current sha256. |
403 SignatureDoesNotMatch |
The body size differs from sizeBytes, a listed header (including X-Amz-Checksum-Sha256) is missing or different, or the upload used chunked transfer. |
Fix the request. If the file changed after you measured it, request a new URL for it. |
403 AccessDenied with message Request has expired |
The URL is past expiresAt. |
Request a new URL for that file and upload again. |
403 AccessDenied (any other message) |
SAFE storage refused the upload. | Do not retry. Contact SAFE support with the uploadId and upload time (UTC). |
Warning
An upload URL is a secret. Anyone who has it can upload only the exact file it was issued for (same size and SHA-256), once, until it expires, without signing in. It cannot upload any other content, or read, list, delete or overwrite anything. Do not log, store or share upload URLs.
Reference script: push files to SAFE
An end-to-end Python 3 example of Steps 2 to 4. It writes signals to gzipped NDJSON (write_signals), checks each file is within the size limit, computes each file's sha256, requests upload URLs 2 files at a time (the API accepts up to 10; a small group finishes uploading well before its URLs expire), uploads each file with exactly the headers SAFE returned, and records each file's uploadId in safe-uploads.csv.
- On
429it waits the seconds inRetry-After; on5xxit retries with exponential backoff; on401the client authenticates again. - A file whose upload fails with
403needs a new upload URL: run the script again for that file. A400 BadDigestmeans the file changed after it was hashed; run the script again for it. - It never logs access tokens or upload URLs.
Show the script: safe_push.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 |
|
What happens after you upload
- A
200on the upload is your confirmation. There is no separate status API: SAFE validates and processes the file in the background. - SAFE matches every asset to the assets it already knows (see Asset identity in Step 2) and applies the full-snapshot rule per source.
- Lines that cannot be read, or signals SAFE cannot identify, are skipped; the rest of the file is still processed. A file that is not gzip, or is corrupt gzip, is rejected whole.
- The data appears on the Assets and Findings pages after processing. In the Asset Source column, data lake sources show the data lake icon; hover over it to see the source name. Use the Asset Source filter to show one source.
| Situation | What you see in SAFE |
|---|---|
| Qualys and Tenable both report server WEB-01 (each in its own files) | One asset, WEB-01, listing both sources. |
| Both report CVE-2023-44487 on it | The vulnerability is kept once per source on that one asset. |
| The same software appears with a new version | The new version is added; the old one is marked as gone when it stops being sent. |
| A vulnerability is left out of Qualys's next signal for WEB-01 | The vulnerability is closed for Qualys on WEB-01. |
Check the status of an upload
- A
200on the upload means SAFE has the file. Processing runs in the background and usually takes minutes; a large push takes longer. - To confirm the data arrived, open the Assets or Findings page and filter on your source name (Asset Source or Finding Source).
- There is no status API today. Keep each file's
uploadId(the reference script writes them tosafe-uploads.csv). If data does not appear, send SAFE support theuploadIdand upload time (UTC): for each upload, support can see whether it was processed, how many assets were read, sent and skipped, and why each skipped line was skipped.
Findings and controls SAFE does not recognise yet
Every finding and control result you send is stored and shown on the Findings page under your source name, whether or not SAFE already knows it. Source tools SAFE has no connector for are accepted too: the name you send in entity.source is the name SAFE shows and filters on.
SAFE uses findings in two ways, and only the second depends on SAFE recognising the finding:
- Finding score and exposure score. Every vulnerability and failed control gets a SAFE finding score and severity, and counts towards the exposure score of its asset and of the asset's groups. For a CVE, SAFE scores it from its own vulnerability intelligence. For a finding SAFE does not recognise, the score is an estimate based on the description and severity you send, so it is less precise.
- Risk scenarios (likelihood and loss). A finding counts towards risk scenarios only when SAFE has mapped it to its control framework (FAIR-CAM). SAFE's Threat Intel team maintains this mapping.
| What you send | Finding score and exposure score | Risk scenarios (likelihood and loss) |
|---|---|---|
Vulnerability whose name is a CVE ID (for example CVE-2024-6387) |
Yes | Yes. CVEs are mapped once for every tool, so they count from any source. |
| Any other vulnerability, and configuration control results | Yes (estimated if SAFE does not recognise the finding) | Only after SAFE has mapped the finding or control. Your SAFE representative can tell you which of yours are already mapped. |
Getting your findings and controls mapped
- Before your first push, and whenever your tools add new checks, send your SAFE representative the list of findings and controls your sources can produce: source name, the finding or control
nameexactly as you send it,type(vaorca),controlId,policyIdorpolicyName, severity, and the description and remediation text. - SAFE's Threat Intel team maps them to its controls and adds them to SAFE.
- Your next regular push re-sends the findings; from then on they also count towards risk scenarios.
Keep finding and control names stable: a reworded name is a new finding and needs mapping again.
Best practices
- Keep each source together. Send each source's assets in as few, full files as possible, and within a file keep each source's rows together rather than mixed. One file per tool is the simplest way.
- Batch assets. Put many assets in each file, and request upload URLs only for as many files as you can upload before they expire (10 minutes): the reference script asks for 2 at a time; the API accepts up to 10. Avoid one asset per file or one push per asset. At the current rate limit a tenant can request 10 upload URLs a minute, so prefer fewer, larger files (up to 250 MiB each).
- Do not upload the same asset twice at once. Do not upload two files that contain the same asset and source at the same time: the order in which they are processed is not guaranteed.
- Send identity well. Always send
cloudResourceIdfor cloud resources andmacAddresswith generic host names. Keep host names stable between pushes; renaming a device can create a second asset. - Keep source names and finding text stable. Changing a source name creates a new source; rewording the name of a non-CVE finding creates a new finding.
- Validate before uploading. Check that every line is valid JSON, has the envelope fields,
entity.sourceand an identity field, and uses the JSON types in the field reference. - Handle errors. Re-authenticate once on
401; on429wait forRetry-After; on5xxretry with exponential backoff; do not retry other4xxerrors unchanged. - Protect secrets. Keep API credentials in a secrets manager and never log access tokens or upload URLs. Log the
uploadIdand file name instead.
Errors and troubleshooting
Upload URL API (POST /api/v3/integrations/file-ingest/uploads)
| Status | Cause | Fix |
|---|---|---|
400 |
The body is not {"files": [{"sizeBytes": N, "sha256": "..."}, ...]}, there are 0 or more than 10 files, a sizeBytes is out of range, or a sha256 is missing or not the base64 SHA-256 of the file. The message names the file, for example files[1].sizeBytes must be between 1 and 262144000 or files[0].sha256 must be the base64-encoded SHA-256 of the file (44 characters). |
Correct the body. Split large files. |
401 |
Expired or invalid access token | Authenticate again (Step 1) and retry. |
403 file ingest is not enabled for this tenant |
The API Connector is not enabled for your tenant | Contact your SAFE representative. |
403 (other) |
No access token was sent; the user does not have permission to manage integrations; or the calling IP address is not in your tenant's allowed IP list | Send the token; use API credentials of a user with that permission; or add the IP address to your allowed list. |
429 |
upload URL limit reached for this tenant; retry after N seconds: your tenant has used its upload URLs for the current minute. A 429 without Retry-After is a general API request limit, separate from the upload URL limit. |
Wait the number of seconds in Retry-After (back off exponentially if it is absent), then retry. A request for fewer files may succeed sooner. |
5xx |
Temporary SAFE error; no URLs were returned | Retry with exponential backoff. |
FAQs
Where can I find SAFE’s APIs?
All SAFE APIs, including POST /api/v3/auth and POST /api/v3/integrations/file-ingest/uploads, are listed in SAFE’s Swagger documentation. See Accessing SAFE APIs in the SAFE documentation for how to open it and create API credentials.
How long is an access token valid, and how do I renew it?
Sixty minutes. Reuse one token for all calls in that time and authenticate again shortly before it expires or when a call returns 401. The reference authentication script does both.
Is there an example script for the whole push?
Yes: see Reference script: push files to SAFE. It covers authentication, building and measuring files, requesting upload URLs, uploading and logging each uploadId.
What version should my records use?
Every line must carry "version": "3.0.0". A line with any other version, including the earlier "2.0.0", is skipped.
How often should I push data?
Push every asset once to load SAFE, then push changed assets on a schedule, typically once every 24 hours. Each asset you send must be its full, current picture for that source (see The full-snapshot rule).
How do I close a vulnerability or mark a control as passed?
Leave the vulnerability out of the source’s next signal for that asset, or send it with status.complianceStatus pass. Send a control as fail or pass.
Can I remove or expire findings?
Yes, by leaving them out: anything SAFE had from a source for an asset that is missing from the source’s next signal for that asset is marked as gone. Assets you do not send are not changed.
How does SAFE know whether an asset is new or one it already has?
It matches on cloud resource ID, then host name, then MAC address, then IP address. See Asset identity: how SAFE recognises a device.
I uploaded data from several tools for the same server. Will I see duplicates?
No. Send each tool under its own source name; SAFE merges them into one asset that lists every source.
Can I send data from a tool SAFE does not integrate with?
Yes. Use any source name; SAFE shows it as sent. Its findings get a finding score and count towards exposure scores; they count towards risk scenarios once SAFE has mapped them (see Findings and controls SAFE does not recognise yet).
Why does a finding not change my risk scenarios?
SAFE has not mapped it to its controls yet. It still has a finding score and counts towards exposure scores, but it counts towards risk scenarios (likelihood and loss) only once mapped. CVE vulnerabilities are mapped automatically; for other findings and controls, send your SAFE representative the list your sources produce (see Getting your findings and controls mapped).
I submitted assets but some are not in SAFE. Why?
The usual causes: the line has no identity field, is not valid JSON, has the wrong version, or names no source; the asset matched an existing asset (see Asset identity); it has more findings than the per-asset limit; or the whole file was rejected because it was not gzip or was corrupt gzip. Lines that cannot be read are skipped and the rest of the file is still processed. Send SAFE support the uploadId: they can see why each line was skipped.
How do I check that an upload was processed?
See Check the status of an upload.
I got a 429. What should I do?
Wait the number of seconds in the Retry-After header and retry. Your tenant can request a limited number of upload URLs per minute (see Limits), so prefer fewer, larger files.
What are the common error responses?
400: the request body is wrong, a file size is out of range, or a sha256 is missing or invalid. 401: the token expired; authenticate again. 403: the API Connector is not enabled, the user lacks permission, or the IP address is not allowed. 429: rate limited; wait for Retry-After. 5xx: temporary SAFE error; retry with backoff. See Errors and troubleshooting for each case.
Did this article help?
Let us know by leaving a star rating at the top of this article.
