| Age | Commit message (Collapse) | Author |
|---|
|
Add the migration file for the new Error Log class and
'affect-components' fields.
Signed-off-by: david.reyna@windriver.com <david.reyna@windriver.com>
|
|
* Add "Affected Components" to VUL/INV/DEF, with
automatic inheritance on creation from respective
parent records. With this is a one-time fixup
routine to populate these new fields:
./bin/common/srtool_utils.py \
--fix-inherit-affected-components -f
* Add "Error Log", to capture internal errors in a
formal table and view.
Management > Maintenance > Error Logs
* Sort the Product list when add product investigation
links to Vulnerability records.
* Add Reports for Notification and Error Log
* Other small fixes
Signed-off-by: david.reyna@windriver.com <david.reyna@windriver.com>
|
|
The model for vulnerabilities and investigation is more
stringent in defining the foreign keys when creating
records. Set these values when creating a record to
avoid a 'null-constraint' error.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
When creating a new Vulnerability or Investigation, inherit the
parent object's (CVE/Vulnerability) "comments" field.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Normally the Table display feature caches served pages for
faster refresh. However, in the SRTool the data is very
volaile so this often shows stale information. For now,
disable the caching.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
1.Fix NIST modified pre-emption
* When a CVE appears in the 'Modified' source
* Remove the CVE link to the normal source
* Add the CVE link to the modified source
* When a CVE disappears from the 'Modified' source
* Remove the CVE link from the modified source
* Restore the CVE link to the normal source
* If forcing a normal source update, first gather
the CVEs in the current "Modified" source, and
ignore them when scanning the normal source. This
is to avoid regressive updates.
* Add tracing to help validate this workflow.
2. Fix sql_cve_query() to always return a valid cve_id
even when there are no updates, to avoid adding
cvesource mappings to '-1'. This addresses one of the
issues in '--find-bad-links'.
3. In the NIST details web page, force the display of
impact and exploit scores to two decimal places, to
normalize the current NIST feeds that are outputting
8+ decimal places.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
1. Fix the NIST 'alt-source' routine to correctly use the current 'modified'
datasource to preempt the regular datasources, and also set the CVE
datasource links accordingly.
2. Update the CVE NIST improve Score/Severity repair, also insure
that the 'modified' datasource values pre-empt the regular
datasource values. Also, fix missing and/or obsolete NIST datasource
references for CVEs.
./bin/common/srtool_utils.py --fix-severity [ALL|"NIST Modified Data"|...]
3. Add/improve helper routine to list Score/Severity values across the
many NIST data sources (e.g. modified and regular), plus the the
current CVE values and the current CVE datasource links. This is used
to investigate and validate the above repair routine.
./bin/nist/srtool_nist.py -S CVE-2020-7470
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
In the summary publish report, leave the product cell blank if there
is no Investigation.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Add the generic CVE summary report across products to the core
report list.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Add CVE publishing features, specifically add a method to generate CVE status
across the releases, filterable by CVE status.
Add dynamic schema calculations for the backup database snapshots, to enable
difference scanning even when the schema has been reordered after a migration.
Add first part of database difference scanning code migration.
Fix MITRE scanning for new source files.
[YOCTO #13734]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
srtool: cumulative deployment features and fixes
High level new features:
* Publishing support to external/public databases
* Ability to label products as "active", "inactive", "under development"
Inactive (EOL) products appear but
* Do not affect status propagation
* Do not auto-create defects
Development product status is not exported to pubic database
* Extend NIST download range to 2002..2019
* Added MITRE downloads to provide RESERVED tracking
* Extended audit history tracking and meta-data
* Delete CVE records
* Ability to do "OR" searches (default is "AND")
Example: "CVE-2019-20095 OR CVE-2019-20096 OR CVE-2019-19977"
* Automated defect creation (Jira)
If selected, creates customer defect for selected and active products
Reuse existing defect if present for given product
* Many small sorting, readability, edge case fixes
Backups:
* Add meta-data stamp file for each backup
* Save daily backups with day name instead of day number
* Preserve file dates when making copies to backup
* Add list command
Automated Updates:
* Fix report format
* Add trial run test
Utilities:
* Add 13 new database fix up procedures
Some are one-shot historical fixes, some are learned validation checks
Database Schema:
* Add "SRTool" class to wrap shared enumerations (e.g. Priority)
* Add "Update" class to tag and track audit trail objects
* Change Priority naming to match CVE model instead of JIRA
* Add srt_created/srt_updated to CVE/Vul/Inv/Notify for improved updating and auditing
* Add to Defect the SRT versions of Status, Priority, Outcome
To distinguish these from the customer's defect system's values
Common Tools:
* Fix new CVE auto-scoring to skip CVE's already scored (though still NEW)
* Add automated propagation of Defects/Investigations status to parent Vulnerabilities
See "srtool_common.py" for rule details
CVEs:
* Add MITRE as an automatic upstream source
This is to specifically capture all of the "RESERVED" CVE enumerations which
will not appear in the MIST databases, and have the CVE records in place for
internal investigations and transitions to "public" status.
* Spell out the command arguments in the NIST data source files for greater legibility
* Change Priority naming to match CVE instead of JIRA
* Add parallel status states for "inactive" products
This specifically blocks state propagation from inactive objects to active objects
NIST management script:
* Refactor file for greater clarity
* Reorder methods to reflect workflow order
* Fully spell out names of objects
* Remove temporary holding class "CVE" in favor of dictionary objects
* Debugging enhancements
* Incremental update commands for stepped debugging
For example, ability to fetch/update specific CVE(s)
* Additional debugging flags
[YOCTO #13734]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Fix the key for the product filter in the investigations table,
and the output for full reports in the CVE->defects report.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Add a report lists the Vulnerabilities, Investigations, and Defects
(plus status) for the givne list of CVEs.
Example:
1) Open the CVEs table
2) Search for this string: "CVE-2017-5715 OR CVE-2017-5753 OR CVE-2017-5754"
3) Click Export
* Select the new report "CVE to Defects Table". You can leave the
rest of the settings alone.
* Click "Generate and Download Report"
* Open the report CSV file in your text editor or in Excel.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
In the Investigation page, explicilty list the attached defects and
their status (and controls) for clarity and simplicity.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Enhance the search query processing:
* By default, space separators are treated as AND
* Add the explicit keywords OR and AND
Note: these keywords switch the default separator
* Allow OR searches in addition to AND searches
Note: mixed searches use the SQL precedence rules
* Allow a '-' prefix to exclude a string
* Allow single and double quotes for exact matches
Examples:
* abd def (equals: abc AND def)
* abc OR def (equals: OR abc def)
* abc OR "def ghi" (equals: abc OR 'def ghi')
* abc OR def AND -ghi
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Add these features:
* Allow attaching multiple Vulnerabilities to a CVE
* Allow attaching existing Vulnerability to a CVE
* Allow attaching multiple triaged CVEs to a Vulnerability
Revert change to to execute_process. Some routines that use this
method already apply a "decode()", and you cannot do two decodes
on an object.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Add time till next update in update verbose output.
Indicate in table display when a datasource has an
update policy but no update script.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Various updates and fixes:
* Use the new SRT_EMAIL_* variable names
* Fix hardcoded value for 'from' address
* Add additional error handling
* Allow the email settings to be defined in SrtSetting values, and provide
example in the ACME datasource file
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
|
|
* Add lastUpdatedDate to track when data source was updated
* Leave lastModifedDate to track upstream dates
* Introduce DataSource.DATETIME_FORMAT and ORM.DATASOURCE_DATETIME_FORMAT
to enforce date formatting in the lib and bin code
* Explicitly set 'nocache' for the data source page, so the
refresh will always show the latest
[YOCTO #13131]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Clean up after pull from RBurton pylint updates
* Protect against missing CVE lookup call
* Protect against disabled defect tool when creating defects
* Repair CVE 'score_date' data field default
* Update tool typos for formatting
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
|
|
|
|
|
|
- Fix whitespace
- Remove redundant imports
- Remove redundant semicolons
- Remove redundant try/except
- Remove redundant super()
|
|
|
|
Add backgroup data source updates cleanup
* Add robust method for scanning ORM string lists
* Fix DATASOURCE_FREQUENCY_STR ORM string list
* Adjust '--list' columns
[YOCTO #13131]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Complete the support for backgroup data source updates:
* Add cron-start,cron-stop to srtool_update
* Have cron update run as a user space script to avoid sudo
* Hook cron-start,cron-stop into srt start,stop
* Add list command to show update sources
* Have force command propagate to update script calls, and
add force option to all source scripts
* Add 'srt manage update ...' for access to the update functions
* Add flag SRT_SKIP_AUTOUPDATE and srt option noautoupdate to
disable the automatic update app for development assistance
Related Fixes:
* Set the schema generator to always update on startup (13138)
* Fix CVE 'recommend' default to the integer zero (13139)
with auto-fix at startup for existing databases
[YOCTO #13131]
[YOCTO #13138]
[YOCTO #13139]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Fix the data source update mechanism:
* Move the update functions to "bin/common/srtool_update.py"
* Remove 'lastModifiedDate' from the data source JSON files (since
every restart overwrites any updated values)
* Change the 'update_time' field to a dictionary of offset values
e.g. "{\"weekday\":\"6\",\"hour\":\"2\"}" = day of week, hour of day
* Implement the update frequency calculations
* Implement data source name filters for selected manual updates
* Add a log status file
[YOCTO #13131]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
The initial implementation of passing CVE references used ';' as a separator.
However, some URLs use this charater to include git branch information, for
example:
http://git.qemu-project.org/?p=qemu.git;a=commitdiff;h=4c65fed8b...
Changing the separator characted to a tab fixes this and other unexpected
characters.
[YOCTO #13121]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Add pulldown to set a new defect's priority and components in
in the Investigation screen.
Clean up the data passing from the srtool_defect* call.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
The CVE 'resource' and 'source' values for the CVE references are now
scanned and displayed.
* The JSON scanning has been moved away from CveResources to a dynamic
value in the CveDetail record, similar to the CPE table processing.
* Additional debugging support has been added
* The now unused CveResources table will be deleted in a later revision
[YOCTO #13121]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Change the new defect call to use named parameters. This will
enhance the readability and better allow for future changes.
Also, pass the CVE list and defect 'reason' so that the defect
integation tool can use that for the defect record and/or
use in creating its own version of the defect 'summary'.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Enable the feature of creating defects from investigations. Consolidate
into one defect creation method for both investigations and CVE
triage.
Enhance the "srtool_defect.py" sample tools to simulate creating
new defects.
Fix the sample "srtool_jira.py" tool new defect creation to support the
new "defect_tag" variable.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
The 'toaster_render' was intended to define global context values.
That feature is better provided by the existing 'managedcontextprocessor'.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Provide an example in ACME on how main apps can extent or replace
existing reports.
This example adds a new report "ACME Product Summary" type to the
existing Product page export/report command.
Also, fix defects in existing report.py.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Update the 'urlpatterns' processing to use the master app.
Also, update the YP master app to include a url and view class, plus
provide a default YP landing page, and abtract the default logo display.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Transition the datasource scanning from 'datasource_org' to
the new master app environment variable, so that it all works
off of one key.
Also, add a sample logo for ACME, plus fix datasource trace details.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
The SRTool allows users to substitute an alternate master application
instead of the default "yp" in order to customize their instance to
their organization.
This is done by:
(a) Creating a datasource directory under bin
(b) Defining a "datasource.json" file
(c) Defining 'export SRT_MAIN_APP="<app>"' in "srtool_env.sh"
This environment files are scanned by 'bin/srt', and if such an
alternate master app is found it pre-empts the default 'yp'.
This value is set via the environment because "lib/srtmain/settings.py"
is the file that sets the app (and this the URL) ordering, and it is
processed before any database is attached.
To disable the alternate main app, simply rename its "datasource.json"
file and it will be ignored for the next start.
The sample alternate app "acme" is provided to demonstrate this facility.
Additionally, a development tool 'bin/dev_tools/master_app.sh' has been
added to help switch between master apps, to aid testing.
$ ./stop.sh
$ ./master_app.sh acme
$ ./start.sh
... test ...
$ ./stop.sh
$ ./master_app.sh yp
$ ./start.sh
Other included fixes:
* Fix the ACME JSON files formating
* Remove ACME "_sample" from all but "datasource.json_sample"
* Fix tabs to spaces in "srt"
* Add global contect values to views::managedcontextprocessor so
that other app templates can share them
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
The functionality was moved to the more flexible
'datasource.json' files.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
* Move the "Fetch Alt Sources" out of the authenticated user block
* Connect "Documentation" to the new User wiki page
* Minor typos and debugging line fixes
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
[YOCTO: 13099]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
[YOCTO #13093]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Fixes:
* Support Django development head in version check
(e.g. '2.2.dev20181217100344')
* Remove the single quotes around the comments content
* Include Documentation/Export links for Guest users
* Allow 'ip:port/acme' to link to 'acme_hello'
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
Remove the app/uls scanning code in 'srtmain/urls.py' in
favor of a fixed deterministic app (and thus URL) ordering.
This will insure that any templates added to the custom app
(e.g. 'acme') will superceed content in 'srtgui', and anything
in srtgui will superceed anythinging in 'users'.
Signed-off-by: David Reyna <David.Reyna@windriver.com>
|
|
|
|
|
|
|
|
|
|
|
cody.yu@windriver.com
David Reyna
Ross Burton