AgeCommit message (Collapse)Author
2020-02-02srtool: add 'affected-components' and ErrorLog migrationHEADmastercody.yu@windriver.com
Add the migration file for the new Error Log class and 'affect-components' fields. Signed-off-by: david.reyna@windriver.com <david.reyna@windriver.com>
2020-02-02srtool: extend 'Affected Components', add Error Logcody.yu@windriver.com
* Add "Affected Components" to VUL/INV/DEF, with automatic inheritance on creation from respective parent records. With this is a one-time fixup routine to populate these new fields: ./bin/common/srtool_utils.py \ --fix-inherit-affected-components -f * Add "Error Log", to capture internal errors in a formal table and view. Management > Maintenance > Error Logs * Sort the Product list when add product investigation links to Vulnerability records. * Add Reports for Notification and Error Log * Other small fixes Signed-off-by: david.reyna@windriver.com <david.reyna@windriver.com>
2020-01-31srtool: enhance database utility validation and repair functionsDavid Reyna
1. Support analyzing alternate databases (e.g. backups) with potentially older schemas by autogenerating and loading respective "srt_schema.py" files, using the '--database /path/to/srt.sqlite' option. 2. Add the following reports to help analyize the distribution of V3/V2 severity values across the CVE years and across the CVE/VUL/INV records. This helps spot trends and potential translation errors. ./bin/common/srtool_utils.py --report-cve-status-summary ./bin/common/srtool_utils.py --report-db-status-summary 3. Add a report to display VUL/INV/DEF recoords that are unattached to any parent CVE/VUL/INV records respectively. ./bin/common/srtool_utils.py --report-unattached-records 4. Update the repair routines to report issues, but only fix them if the "--force" flag is set. This allows the review of the potential fixes before committing them. 5. General clean and internal documentation. Rename the commands to better distinguish "fix" (one-shot), "repair" (on-going), and "report" functions. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-31srtool: preset foreign keys in object creationDavid Reyna
The model for vulnerabilities and investigation is more stringent in defining the foreign keys when creating records. Set these values when creating a record to avoid a 'null-constraint' error. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-26srtool: inherit comments to new vulnerabilitys and investigationsSRTOOL_DJANGO_1_11David Reyna
When creating a new Vulnerability or Investigation, inherit the parent object's (CVE/Vulnerability) "comments" field. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-26srtool: disable table display cachingDavid Reyna
Normally the Table display feature caches served pages for faster refresh. However, in the SRTool the data is very volaile so this often shows stale information. For now, disable the caching. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-26srtool: fix NIST modified pre-emption, bad cvesource mappingsDavid Reyna
1.Fix NIST modified pre-emption * When a CVE appears in the 'Modified' source * Remove the CVE link to the normal source * Add the CVE link to the modified source * When a CVE disappears from the 'Modified' source * Remove the CVE link from the modified source * Restore the CVE link to the normal source * If forcing a normal source update, first gather the CVEs in the current "Modified" source, and ignore them when scanning the normal source. This is to avoid regressive updates. * Add tracing to help validate this workflow. 2. Fix sql_cve_query() to always return a valid cve_id even when there are no updates, to avoid adding cvesource mappings to '-1'. This addresses one of the issues in '--find-bad-links'. 3. In the NIST details web page, force the display of impact and exploit scores to two decimal places, to normalize the current NIST feeds that are outputting 8+ decimal places. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-24srtool: fix alt-source updates, improve CVE repairDavid Reyna
1. Fix the NIST 'alt-source' routine to correctly use the current 'modified' datasource to preempt the regular datasources, and also set the CVE datasource links accordingly. 2. Update the CVE NIST improve Score/Severity repair, also insure that the 'modified' datasource values pre-empt the regular datasource values. Also, fix missing and/or obsolete NIST datasource references for CVEs. ./bin/common/srtool_utils.py --fix-severity [ALL|"NIST Modified Data"|...] 3. Add/improve helper routine to list Score/Severity values across the many NIST data sources (e.g. modified and regular), plus the the current CVE values and the current CVE datasource links. This is used to investigate and validate the above repair routine. ./bin/nist/srtool_nist.py -S CVE-2020-7470 Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-23srtool: fix routine for broken V3/V2 status in CVE recordsDavid Reyna
Add a fixup utility to repair error from the MITRE CVE creation script that left broken V2 status values. Also add a NIST status summary debug command to report the CVE general status across the base source file, the 'modified' source file, and list the current datasource mappings for that CVE. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-22srtool: fix score_date formatDavid Reyna
The schema for this field is 'models.DateField' but the scoring method in "srtool_common --score-new-cves" was setting an obsolete date_time value. That crashes Django-2.2 (but not Django-1.11). Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-21srtool: publish leave product empty if not investigationDavid Reyna
In the summary publish report, leave the product cell blank if there is no Investigation. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-21srtool: add CVE summary report to core reportsDavid Reyna
Add the generic CVE summary report across products to the core report list. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-21srtool: add publishing, update backup scanning, MITRE init fixDavid Reyna
Add CVE publishing features, specifically add a method to generate CVE status across the releases, filterable by CVE status. Add dynamic schema calculations for the backup database snapshots, to enable difference scanning even when the schema has been reordered after a migration. Add first part of database difference scanning code migration. Fix MITRE scanning for new source files. [YOCTO #13734] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-12srtool: add MITRE 2020David Reyna
Add the MITRE 2020 data source [YOCTO #13734] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2020-01-12Bug 13734 - cumulative deployment features and fixesDavid Reyna
srtool: cumulative deployment features and fixes High level new features: * Publishing support to external/public databases * Ability to label products as "active", "inactive", "under development" Inactive (EOL) products appear but * Do not affect status propagation * Do not auto-create defects Development product status is not exported to pubic database * Extend NIST download range to 2002..2019 * Added MITRE downloads to provide RESERVED tracking * Extended audit history tracking and meta-data * Delete CVE records * Ability to do "OR" searches (default is "AND") Example: "CVE-2019-20095 OR CVE-2019-20096 OR CVE-2019-19977" * Automated defect creation (Jira) If selected, creates customer defect for selected and active products Reuse existing defect if present for given product * Many small sorting, readability, edge case fixes Backups: * Add meta-data stamp file for each backup * Save daily backups with day name instead of day number * Preserve file dates when making copies to backup * Add list command Automated Updates: * Fix report format * Add trial run test Utilities: * Add 13 new database fix up procedures Some are one-shot historical fixes, some are learned validation checks Database Schema: * Add "SRTool" class to wrap shared enumerations (e.g. Priority) * Add "Update" class to tag and track audit trail objects * Change Priority naming to match CVE model instead of JIRA * Add srt_created/srt_updated to CVE/Vul/Inv/Notify for improved updating and auditing * Add to Defect the SRT versions of Status, Priority, Outcome To distinguish these from the customer's defect system's values Common Tools: * Fix new CVE auto-scoring to skip CVE's already scored (though still NEW) * Add automated propagation of Defects/Investigations status to parent Vulnerabilities See "srtool_common.py" for rule details CVEs: * Add MITRE as an automatic upstream source This is to specifically capture all of the "RESERVED" CVE enumerations which will not appear in the MIST databases, and have the CVE records in place for internal investigations and transitions to "public" status. * Spell out the command arguments in the NIST data source files for greater legibility * Change Priority naming to match CVE instead of JIRA * Add parallel status states for "inactive" products This specifically blocks state propagation from inactive objects to active objects NIST management script: * Refactor file for greater clarity * Reorder methods to reflect workflow order * Fully spell out names of objects * Remove temporary holding class "CVE" in favor of dictionary objects * Debugging enhancements * Incremental update commands for stepped debugging For example, ability to fetch/update specific CVE(s) * Additional debugging flags [YOCTO #13734] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-20srtool_reports: fix investigations product filter, CVE reportDavid Reyna
Fix the key for the product filter in the investigations table, and the output for full reports in the CVE->defects report. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-20srtool_update: improve update reporting and schedulingDavid Reyna
Track the current running update task in ".srtupdate.task" to help track background update activity and overhead. Make calls to the update start/stop absolute paths to help track active SRTool tasks, especially between multiple servers. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-20srtool_common: fix package updatesDavid Reyna
Fix a misplaced ')' in the updated package registration code. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-19srtool_jira: fix Jira status mappingsDavid Reyna
Fix a copy/paste error in the Jira status mapping table. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-19srtool_update: fix product/user update from tablesDavid Reyna
When restarting the SRTool, the main app's user and product table should be re-read and applied. This fixes a select problem. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-19srtool_report: add CVE to defects reportDavid Reyna
Add a report lists the Vulnerabilities, Investigations, and Defects (plus status) for the givne list of CVEs. Example: 1) Open the CVEs table 2) Search for this string: "CVE-2017-5715 OR CVE-2017-5753 OR CVE-2017-5754" 3) Click Export * Select the new report "CVE to Defects Table". You can leave the rest of the settings alone. * Click "Generate and Download Report" * Open the report CSV file in your text editor or in Excel. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-09srtool_srt: add 'manage' usage help displayDavid Reyna
Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-09srtool_investigation: list investigation defects and statusDavid Reyna
In the Investigation page, explicilty list the attached defects and their status (and controls) for clarity and simplicity. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-05srtool_utils: add master app clone supportDavid Reyna
Update the master app tool to create new master apps. Also, update the master app change feature to remove the previous apps data source entries. Note that the existing users and products are left untouched (to keep the database records working); such such obsolete content must be removed manually or the data base restarted clean. Creation example, using 'yoyo' for the "Yoyodyne Corporation": $ ./stop.sh $ ./master_app create yoyo $ ./start.sh Switch bask to Yocto Project example: $ ./stop.sh $ ./master_app yp $ ./start.sh Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-02-03srtool_search: enhance search with OR, Quotes, ExcludeDavid Reyna
Enhance the search query processing: * By default, space separators are treated as AND * Add the explicit keywords OR and AND Note: these keywords switch the default separator * Allow OR searches in addition to AND searches Note: mixed searches use the SQL precedence rules * Allow a '-' prefix to exclude a string * Allow single and double quotes for exact matches Examples: * abd def (equals: abc AND def) * abc OR def (equals: OR abc def) * abc OR "def ghi" (equals: abc OR 'def ghi') * abc OR def AND -ghi Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-31srtool_cve: add multiple cve attach to a vulnerabilityDavid Reyna
Add these features: * Allow attaching multiple Vulnerabilities to a CVE * Allow attaching existing Vulnerability to a CVE * Allow attaching multiple triaged CVEs to a Vulnerability Revert change to to execute_process. Some routines that use this method already apply a "decode()", and you cannot do two decodes on an object. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-31srtool-updates: improve update reportingDavid Reyna
Add time till next update in update verbose output. Indicate in table display when a datasource has an update policy but no update script. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-30srtool-mitre: implement CVE NEW_RESERVED statusDavid Reyna
Update the Mitre scanning tool to convert recent (and/or no date) reserved CVEs to the state NEW_RESERVED. This will keep them separate from the new CVEs that need triage, plus keep them from being unnecessarily scored by the background process. Add a fixup rountine for previoulsy imported databases: $ ./bin/common/srtool_utils.py --fix-new-reserved Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-29srtool_email: update email values, error handlingDavid Reyna
Various updates and fixes: * Use the new SRT_EMAIL_* variable names * Fix hardcoded value for 'from' address * Add additional error handling * Allow the email settings to be defined in SrtSetting values, and provide example in the ACME datasource file Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-24srtool: separate name/passwords for defect and emailDavid Reyna
Separate the environment variables of the username and password for the defect and email systems into separate values, in case they need separate credentials. Also, fix a Wind River-ism in the Jira template to reflect that the product key is not necessarily also defect name prefix. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-22srtool_README: update the git repo README fileDavid Reyna
Add the new web links and mailing list. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-21srtool_start: generate SRTool stop/restart scriptsDavid Reyna
When the SRTool successfully starts, auto-generate helper scripts to allow the user to stop and restart the SRTool based on the previous options. This is also useful for scripts that need to stop then restart the SRTool server in order to safely perform actions (like backup or restore). Generated scripts: bin/srt_start.sh (also first stops the server if currently running) bin/srt_stop.sh Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-20srtool_backup: simplify the backup scriptDavid Reyna
Reset the backup tool to simply save the (a) sqlite database, the (b) data files, and (c) the attachments. Support both the weekly backup (to "backup_$year_$weeknum") and daily backup (to "backup_$weekday"). The previous JSON export format is being reworked. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-19srtool_patcher: fix in-line change handling in public areaDavid Reyna
Only capture the custom sections in the in-line patch file, so that changes to the public areas are kept to the original file. This allows the user to push those public differences or remove them, and not affect the application of the custom patch file. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-19srtool_patcher: add custom exclusion supportDavid Reyna
Add the ability to exclude sections of the common code. The primary use case is if the common code is executinga function that causes undesired side effects for the customization. Here is example code in "bin/acme/patcher/inplace/bin/srt.patch": ### ACME_EXTENSION_EXCLUDE_BEGIN ### #echo "The system will $CMD." # ### ACME_EXTENSION_EXCLUDE_END ### ### ACME_EXTENSION_BEGIN ### # # NOTE: Exclusions should be used only when necessary # NOTE: The excluded lines are commented so that they are inactive # but restorable # echo "The ACME SRTool system will now $CMD!" ### ACME_EXTENSION_END ### The content in the excluded section are commented, so that when it is cleaned those lines can be restored to the original. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-19srtool: add in-place custom patching supportDavid Reyna
Add 'in-place' patch support for customizing organization patches on top of regular upstream SRTool files. An example is provided in "bin/acme/patcher/inplace/bin/srt", which adds a custom hello message. * To assert (merge) the custom version for runtime, run: $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json -I * To stash the customized version and replace with clean version (but keep any changes in the public sections for pushing upstream), run: $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json -i Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-17srtool: clean up patcher scriptDavid Reyna
Fix the description and usage examples in the header. Remove unused command. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-17srtool: patcher, Jira template, Jira ACMEDavid Reyna
Add formal support for extending and customizing shared template code files. * The initial use case is a shared Jira integration file that partners can extend to their particular installation. A working example is provided in the ACME directory: upstream: "bin/common/srtool_jira_template.py" custom : "bin/acme/srtool_jira_acme.py" * The custom sections are blocked off with comment tags (e.g. ACME) ### ACME_EXTENSION_BEGIN ### ... ### ACME_EXTENSION_END ### * The './bin/common/srtool_patcher.py' tool provides the support, for example: To merge shared upstream code into the custom Jira script, run: $ ./bin/common/srtool_patcher.py -j bin/acme/patcher.json --merge-original To merge edits in script's common areas back to upstream, run: $ ./bin/common/srtool_patcher.py -j bin/acme/patcher.json --merge-custom Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-16srtool: pylint for the bin toolsDavid Reyna
Fix the pylint errors in the bin directory tools. The pylint command was: $ PYTHONPATH=./lib:./bin pylint3 --load-plugins pylint_django bin --disable=C,R,unused-variable,unused-wildcard-import,redefined-outer-name,unused-argument,fixme,bare-except,broad-except,redefined-builtin,unnecessary-pass,logging-not-lazy,wildcard-import | tee srt_bin_pylint.txt The currently allowed exceptions are: W0603: Using the global statement (global-statement) W0611: Unused ORM imported from srt_schema (unused-import) Also, remove the obsolete 'bin/common/srtool_defect.py' and skip the need to pylint it altogether. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-15views: use subprocess directlyRoss Burton
2019-01-15srtool: separate lastUpdatedDate, enforce date formatDavid Reyna
* Add lastUpdatedDate to track when data source was updated * Leave lastModifedDate to track upstream dates * Introduce DataSource.DATETIME_FORMAT and ORM.DATASOURCE_DATETIME_FORMAT to enforce date formatting in the lib and bin code * Explicitly set 'nocache' for the data source page, so the refresh will always show the latest [YOCTO #13131] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-14srtool: updates cleanup #2David Reyna
Clean up after pull from RBurton pylint updates * Protect against missing CVE lookup call * Protect against disabled defect tool when creating defects * Repair CVE 'score_date' data field default * Update tool typos for formatting Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-14orm: pylintRoss Burton
2019-01-14pylint srtmainRoss Burton
2019-01-14pylint srtguiRoss Burton
2019-01-14srtgui: pylint-inspired cleanupsRoss Burton
- Fix whitespace - Remove redundant imports - Remove redundant semicolons - Remove redundant try/except - Remove redundant super()
2019-01-14lib/orm: clean up importsRoss Burton
2019-01-14acme: cleanup importsRoss Burton
2019-01-14srtool-requirements: add requestsRoss Burton
This is used by lib/srtmain/management/commands/perf.py.
2019-01-14srtool: auto-update support cleanupDavid Reyna
Add backgroup data source updates cleanup * Add robust method for scanning ORM string lists * Fix DATASOURCE_FREQUENCY_STR ORM string list * Adjust '--list' columns [YOCTO #13131] Signed-off-by: David Reyna <David.Reyna@windriver.com>