AgeCommit message (Collapse)AuthorFilesLines
2019-01-13srtool: fix core update implementationDavid Reyna21-64/+325
Fix the data source update mechanism: * Move the update functions to "bin/common/srtool_update.py" * Remove 'lastModifiedDate' from the data source JSON files (since every restart overwrites any updated values) * Change the 'update_time' field to a dictionary of offset values e.g. "{\"weekday\":\"6\",\"hour\":\"2\"}" = day of week, hour of day * Implement the update frequency calculations * Implement data source name filters for selected manual updates * Add a log status file [YOCTO #13131] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-12srtool: add missing environment filesDavid Reyna2-0/+6
Add the default environment extension files for 'bin/common' and 'bin/yp'. They are currently passive. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-12srtool: improve CVE status assignments, add CVE 2019David Reyna14-133/+176
1. Add the CVE 2019 data soures for MITRE and NIST. 2. Improve the CVE default status assignment system: * During the "Init" phase all CVEs default to HISTORICAL, unless they are within the CVE_INIT_NEW_DELTA date range. The value CVE_INIT_NEW_DELTA is defined in "bin/common/datasource.json", and is an out-of-box courtesy to provide some CVEs for triage in newly initialized systems. Changing the default value to '0' disabled this. * During the "Update" phase, CVEs default to NEW (and thus primed for triage) * Better separate the Init versus Update functions in "srtool_mitre.py" and "srtool_nist.py", and their respective datasource files. * Remove the post-process "preset_new()" in "srtool_common.py" in favor of directly computing the values in get_cve_default_status() in "srtool_mitre.py" and "srtool_nist.py", for speed and consistency. [YOCTO #13134] [YOCTO #13135] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-08srtool: remove obsolete 'orm_cvereference' table checkDavid Reyna1-1/+0
Remove the obsolete and now empty 'orm_cvereference' table from the sanity check. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-07srtool: fix parsing CVE reference separatorDavid Reyna2-2/+2
The initial implementation of passing CVE references used ';' as a separator. However, some URLs use this charater to include git branch information, for example: http://git.qemu-project.org/?p=qemu.git;a=commitdiff;h=4c65fed8b... Changing the separator characted to a tab fixes this and other unexpected characters. [YOCTO #13121] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-07srtool: fix yp product order numberingDavid Reyna1-4/+5
Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-07yoto-project-products: add 2.7/WarriorRoss Burton1-0/+10
2019-01-07yocto-project-products: fix CPEsRoss Burton1-3/+3
The first number is the CPE version so this should always be 2.3. Fix the Yocto Project release version for Thud to be 2.6 instead of 2.5.
2019-01-07bin/*/srtool: don't try Python 2 importsRoss Burton6-39/+15
2019-01-07srtool: add 'superuser' helper script, improve 'tail'David Reyna2-9/+16
Add a devtool helper script 'suport.sh' to help start the super user setup call. Add 'srt_err.log' to 'tail.sh'. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-07srtool: set priority/components new defect from investigationDavid Reyna2-14/+104
Add pulldown to set a new defect's priority and components in in the Investigation screen. Clean up the data passing from the srtool_defect* call. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-04srttool: include missing CVE reference informationDavid Reyna6-40/+43
The CVE 'resource' and 'source' values for the CVE references are now scanned and displayed. * The JSON scanning has been moved away from CveResources to a dynamic value in the CveDetail record, similar to the CPE table processing. * Additional debugging support has been added * The now unused CveResources table will be deleted in a later revision [YOCTO #13121] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2019-01-01srtool: parameterize the defect new callDavid Reyna4-36/+222
Change the new defect call to use named parameters. This will enhance the readability and better allow for future changes. Also, pass the CVE list and defect 'reason' so that the defect integation tool can use that for the defect record and/or use in creating its own version of the defect 'summary'. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-30srtool: add defect create from investigationsDavid Reyna6-90/+170
Enable the feature of creating defects from investigations. Consolidate into one defect creation method for both investigations and CVE triage. Enhance the "srtool_defect.py" sample tools to simulate creating new defects. Fix the sample "srtool_jira.py" tool new defect creation to support the new "defect_tag" variable. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-30srtool: replace 'toaster_render' with 'managedcontextprocessor'David Reyna1-27/+19
The 'toaster_render' was intended to define global context values. That feature is better provided by the existing 'managedcontextprocessor'. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-30srtool: enable custom report extensionsDavid Reyna4-11/+221
Provide an example in ACME on how main apps can extent or replace existing reports. This example adds a new report "ACME Product Summary" type to the existing Product page export/report command. Also, fix defects in existing report.py. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-30srtool: fix url patterns for master appDavid Reyna8-14/+116
Update the 'urlpatterns' processing to use the master app. Also, update the YP master app to include a url and view class, plus provide a default YP landing page, and abtract the default logo display. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-30srtool: replace 'datasource_org' with new master app codeDavid Reyna7-32/+19
Transition the datasource scanning from 'datasource_org' to the new master app environment variable, so that it all works off of one key. Also, add a sample logo for ACME, plus fix datasource trace details. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-29srtool: generalize the master app (yp, acme, ...) managementDavid Reyna23-35/+392
The SRTool allows users to substitute an alternate master application instead of the default "yp" in order to customize their instance to their organization. This is done by: (a) Creating a datasource directory under bin (b) Defining a "datasource.json" file (c) Defining 'export SRT_MAIN_APP="<app>"' in "srtool_env.sh" This environment files are scanned by 'bin/srt', and if such an alternate master app is found it pre-empts the default 'yp'. This value is set via the environment because "lib/srtmain/settings.py" is the file that sets the app (and this the URL) ordering, and it is processed before any database is attached. To disable the alternate main app, simply rename its "datasource.json" file and it will be ignored for the next start. The sample alternate app "acme" is provided to demonstrate this facility. Additionally, a development tool 'bin/dev_tools/master_app.sh' has been added to help switch between master apps, to aid testing. $ ./stop.sh $ ./master_app.sh acme $ ./start.sh ... test ... $ ./stop.sh $ ./master_app.sh yp $ ./start.sh Other included fixes: * Fix the ACME JSON files formating * Remove ACME "_sample" from all but "datasource.json_sample" * Fix tabs to spaces in "srt" * Add global contect values to views::managedcontextprocessor so that other app templates can share them Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-29srtool: add quick test for python3 and sqlite3David Reyna1-2/+9
Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-29srtool: remove obsolete fixture filesDavid Reyna7-657/+3
The functionality was moved to the more flexible 'datasource.json' files. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-28srtool: allow fetch alt cve data for guests, add user doc linkDavid Reyna4-5/+4
* Move the "Fetch Alt Sources" out of the authenticated user block * Connect "Documentation" to the new User wiki page * Minor typos and debugging line fixes Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-21srtool_nist: fix typo in exceptionRoss Burton1-1/+1
2018-12-21srtool: fix superuser group value in user tableDavid Reyna2-2/+5
[YOCTO: 13099] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-21srtool: fix pylint reported issuesDavid Reyna7-72/+26
[YOCTO #13093] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-21srtool: cummulative fixes 12/21/2018David Reyna4-19/+21
Fixes: * Support Django development head in version check (e.g. '2.2.dev20181217100344') * Remove the single quotes around the comments content * Include Documentation/Export links for Guest users * Allow 'ip:port/acme' to link to 'acme_hello' Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-21srtool: make apps order deterministicDavid Reyna3-208/+26
Remove the app/uls scanning code in 'srtmain/urls.py' in favor of a fixed deterministic app (and thus URL) ordering. This will insure that any templates added to the custom app (e.g. 'acme') will superceed content in 'srtgui', and anything in srtgui will superceed anythinging in 'users'. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-19srtool: fix recreate.sh and sanity test outputDavid Reyna2-4/+4
Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-19report: incorrect class in super()Ross Burton1-1/+1
2018-12-19views: use Django 2's django.urls.resolveRoss Burton1-1/+1
2018-12-19reports: fix variable name typoRoss Burton1-1/+1
2018-12-19srtmain/perf: fix indentationRoss Burton1-34/+34
2018-12-19srtmain: handle debug-toolbar not being installedRoss Burton1-3/+6
2018-12-18srtool: add sanity tool and development toolsDavid Reyna10-0/+347
Run 'bin/common/srtool_sanity_test.py -i' to get a quick sanity test of the database content and the running SRTool server instance. Development helper tools are provided in 'bin/dev_tools' Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-18lib: fix typo in get_name_sortRoss Burton4-4/+4
Ideally, these are all centralised.
2018-12-18srtool_common: use regular expressions to parse sqlite outputRoss Burton1-12/+6
2018-12-18api: correctly decode UTF-8Ross Burton1-5/+1
2018-12-18srtools_utils: remove unused execute_processRoss Burton1-26/+0
2018-12-18lsupdates: remove obsolete importRoss Burton1-2/+1
2018-12-18reports: remove obsolete importRoss Burton1-1/+1
2018-12-18srtool_common: use subprocess directly, decode UTF-8 correctlyRoss Burton1-29/+7
2018-12-18srtool_mitre: use shutil.rmtree to remove directoryRoss Burton1-4/+2
2018-12-18srtool_debian: use shutil.rmtree to remove directoryRoss Burton1-4/+2
2018-12-18srtool: remove obsolete migration fileDavid Reyna1-38/+0
All of the current init/migration are consolidated into the "0001_initial.py" file. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-17srtool: port to Django-2.xDavid Reyna15-128/+172
Support Django-2.2: Move 'django.core.urlresolvers' to 'django.urls' Disable 'register.assignment_tag' tags Move settings 'MIDDLEWARE_CLASSES' to 'MIDDLEWARE' Move urlpatterns 'include' to 'path' Move 'regex.pattern' to 'pattern.regex.pattern' Maintain Django-1.11 support General Fixes: Fix commit for notify_categories Add more error halt checks during lsupdates Add explicit 'on_delete=models.CASCADE' for all ForeignKey's Fix 'get_defect_tag' processing [YOCTO #13091] Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-16srtool: add sample 'acme' product dir, small fixesDavid Reyna15-9/+694
1. Add the 'acme' product directory and URLs, including a sample hello page, a defects toaster table, a products toaster table, and a product table. These can be reached under <IPADD>/acme/hello 2. Small fixes to the CVE selection page. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-14srtool: add sample organization source, add debuggingDavid Reyna9-2/+1431
1. Create the sample "bin/acme_sample" organization data source, to assist companies in adopting and customizing SRTool. 2. Add error detection and halting to the startup datasource scripts. Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-14srtool: remove old verion of srtool requirements fileDavid Reyna1-3/+0
Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-13srtool: fix YP default product paths and formatDavid Reyna2-8/+8
Signed-off-by: David Reyna <David.Reyna@windriver.com>
2018-12-13srtool: cummulative update 12/13/2018David Reyna81-1494/+3259
Changes: Repartition the data sources Reconfigure the data sources into self-contained directories under the "bin" directory. Implement dynamic data source discovery and import Remove all hard coded data source data (e.g. fixtures, data, CVE lookups) Add license files to all data sources Django User model Add "users" Django application dir Login page Self create user account page Password change page User access and delete management CVE Name sorting by hidden 'name_sort' field (CVE-nnnn-0nnnnnn) CVE Triage Auto import reserved CVEs Add MITRE CVE records where NIST missing Add data source count to triage page Easy checkbox toggle by clicking any field Triage any CVE status category (not just new) Assign to any CVE status category Object create/delete Create/Delete Vulnerablities Create/Delete Investigations from Vulnerablity page Add "Historical" CVE status When bootstraping system, all CVEs older than 60 days preset to "Historical" Add CVEs withint 60 days preset to "New" Can be overridden by defect and systaining status imports Preadd Debian data for "New" CVEs Abstraction Add generic Product mappings to defect system ("defect_tag": defect prefix) Add generic Product mappings to product system ("product_tag": product reference, related) Manage functions via "srt" script For example add superuser Normalize Vulnerability to Investigation mapping Replace orm_vulnerabilityproduct with orm_vulnerabilitytoinvestigation General Enable the 'srtool-requirements.txt' Django test Speed the CVE scoring by pre-fetching the datasources Progress display cleanup Move and update srtool_defect prototype to 'bin/yp' Signed-off-by: David Reyna <David.Reyna@windriver.com>