Commit Graph

270 Commits

Author SHA1 Message Date
dvanaken 4d228f5765 Updated task_ids field from CharField to TextField type 2020-02-08 17:31:29 -05:00
dvanaken ec9066e5b2 Updated handling of celery task status in views 2020-02-08 17:31:29 -05:00
dvanaken d9e2806b9e Changes:
- website.celery: create celery app
- website.admin: do not unregister the djcelery models
- website.settings: added some new celery settings and updated others. Added 'celery.tasks' to our logging config which resolved the missing celery log messages issue
- website.async_tasks: removed redundant abstract task classes. Removed the settings that configured retries since do not handle retries in the tasks
2020-02-08 17:31:29 -05:00
dvanaken 7339d07a98 Updated celery package version requirements 2020-02-08 17:31:29 -05:00
dvanaken 06948089d2 Added new model to store execution times 2020-01-23 23:38:56 -05:00
bohanjason 6f0fcfd952 use gpflow in workload mapping 2020-01-22 13:13:32 -05:00
BohanZhang 25d1950e67 use GPFlow in workload mapping 2020-01-22 13:13:32 -05:00
yangdsh 389174302f fix hard coded target objective 2020-01-21 20:02:06 -05:00
Dana Van Aken 35160f2fb4 Upgraded Django from version 1.11.23 to 1.11.27 2020-01-19 06:33:25 -05:00
Dana Van Aken 1d8913f408 When converting a value in bytes to human-readable, if the conversion fails then log a warning and return the original value instead of raising an exception. 2020-01-19 02:43:55 -05:00
yangdsh c2d5cbadd6 rename hyperparameters and use them as defaults 2020-01-18 23:27:01 -05:00
yangdsh cb8c597818 fix format 2020-01-18 23:27:01 -05:00
yangdsh a3774af2c1 modify tests 2020-01-18 23:27:01 -05:00
yangdsh d3c7bb661d fix bugs 2020-01-18 23:27:01 -05:00
yangdsh 24194293bc make session hyperparameters editable on the website 2020-01-18 23:27:01 -05:00
dvanaken 6bf50b892d Moved dumpdata code from resetwebsite into its own mgmt command, ignore errors from stopcelery kill command, added backdoor method to dump website backend database and logfiles 2020-01-14 07:43:31 -05:00
dvanaken 9f9f2845d6 Fixed formatting 2020-01-13 17:21:56 -05:00
dvanaken 7b962c4065 Updated parser classes to not cache knob/metric catalogs and replaced all queries that filter for numeric metrics with the new MetricType.numeric() method. 2020-01-13 17:21:56 -05:00
dvanaken e1a41928f6 Fixed admin forms for Metric/KnobCatalog models. 2020-01-13 17:21:56 -05:00
dvanaken ee101d818e Added class methods to get numeric/nonnumeric metric types. 2020-01-13 17:21:56 -05:00
dvanaken bcad0fa017 Added CHOOSE to enumvals for Oracle parameter optimizer_mode 2020-01-13 10:09:31 -05:00
dvanaken 2f79bfdc17 Fixed parser to treat enum values as case-insensitive 2020-01-13 10:09:31 -05:00
dvanaken 16348d1c7a Updated fixtures, human-readable values values in knob/metric comparison view 2020-01-09 03:23:06 -05:00
dvanaken c33defa29a Updated oracle knob fixtures 2020-01-09 03:23:06 -05:00
dvanaken 4c6ec3f0fd checkpoint 2020-01-09 03:23:06 -05:00
dvanaken 433bb68f2f Added next few Oracle params 2020-01-09 03:23:06 -05:00
dvanaken e1da577ad0 Fixed formatting 2020-01-08 15:29:31 -05:00
dvanaken 9393fb7aca Fixed bugs and improved logging in config recommendation tasks/views 2020-01-08 15:29:31 -05:00
dvanaken 8cec62160f Create/format/save the recommended config for the latest result before the on_success method. 2020-01-08 15:29:31 -05:00
dvanaken 8001b658c9 Added management function similar to 'startcelery' except that it auto-reloads modified website files. 2020-01-07 04:04:06 -05:00
dvanaken 9fced864fe Updated Django settings: removed unused paths, fixed incorrect paths to media/template dirs, increased size of rotating log files from 50KB to 2MB and the number of backup files. 2020-01-03 07:32:28 -05:00
dvanaken 16acc35f54 Fix style 2019-12-21 06:47:48 +01:00
arifiorino d007d932b5 Added target objective to comparison 2019-12-21 06:11:29 +01:00
Bohan Zhang 9633b0e99c improve dnn 2019-12-20 19:08:16 +01:00
yangdsh 3a722df5e2 add missing columns in metrics 2019-12-17 10:02:49 +01:00
yangdsh 0aaa0d37f0 fix target objective for restart failure 2019-12-17 10:02:18 +01:00
dvanaken 2a7bc6145f Fixed db time target objective 2019-12-16 20:47:24 +01:00
yangdsh ade504dcc4 fix bug 2019-12-15 23:09:29 +01:00
yangdsh df86a9f3bc fix website tests 2019-12-15 23:09:29 +01:00
yangdsh 1ebcbf383e regenerate website test fixture 2019-12-15 23:09:29 +01:00
yangdsh 477c8e7682 revert a LOG 2019-12-15 23:09:29 +01:00
yangdsh 7cc0c40d92 isolate workloads of different projects 2019-12-15 23:09:29 +01:00
dvanaken fc4cf0e18b added new oracle knobs 2019-12-15 19:20:29 +01:00
bohanjason 38f44848af disable knob identification 2019-12-15 18:54:03 +01:00
bohanjason ddd97a0973 fix knob identification 2019-12-15 18:54:03 +01:00
bohanjason cae807cf62 add comment 2019-12-15 18:54:03 +01:00
bohanjason b2e530095e clean up 2019-12-15 18:54:03 +01:00
bohanjason d5f7ae65eb disable dummy encoder in knob identification 2019-12-15 18:54:03 +01:00
dvanaken a144069e6b shuffle lhs samples 2019-12-11 19:18:35 +01:00
bohanjason 6d76b22e3b support enum/boolean in GPR/DNN 2019-12-11 09:18:40 +01:00
dvanaken 4b37dec988 Fixed formatting 2019-12-11 00:40:06 +01:00
dvanaken ffc88bfaae Run ddpg++ by default 2019-12-11 00:40:06 +01:00
dvanaken 65f3a7b0d6 Finished adding support for Oracle enums 2019-12-11 00:40:06 +01:00
dvanaken a3cc34ba8e Updated oracle fixtures 2019-12-10 18:01:56 +01:00
dvanaken 42e1a4add3 Make constants editable 2019-12-10 18:01:56 +01:00
yangdsh d1a9933808 fix bug 2019-12-10 17:54:54 +01:00
yangdsh 7ade03e3e5 fix bugs 2019-12-10 10:14:01 +01:00
yangdsh 642d7c3875 fix enum bug 2019-12-10 10:14:01 +01:00
dvanaken a7724233d3 fixed formatting 2019-12-09 22:19:47 +01:00
dvanaken b0cddcc289 Added 3 Oracle hidden knobs 2019-12-09 22:19:47 +01:00
dvanaken 02c40f5cea Adjusted background task time 2019-12-05 20:46:02 -05:00
dvanaken 783b94cd4d Speedup travis-ci tests 2019-12-05 20:46:02 -05:00
dvanaken 9e9c9c7510 Fixed bug when collecting session knobs; for LHS sessions, remove all pre-generated configs if the session knobs are modified 2019-12-05 20:46:02 -05:00
yangdsh 1250732add resolve conflicts 2019-12-05 03:45:18 -05:00
yangdsh 5579d8d94f restore CDBTune 2019-12-05 03:45:18 -05:00
bohanjason 1e17c78956 debug for new gpr 2019-12-05 01:50:41 -05:00
bohanjason 70f9f952d5 hyperparameter debug info for new gpr 2019-12-05 01:50:41 -05:00
bohanjason 7b57eb98a2 add tests for new gpr models 2019-12-04 18:07:36 -05:00
bohanjason 7ebb2ea473 update old gpr model 2019-12-04 18:07:36 -05:00
bohanjason 7ee615a3f3 change default ridge to 1.0 2019-12-04 18:07:36 -05:00
dvanaken 7be5b89975 Changes: when we deploy our docker images we now also build/deploy the internal driver image; fixed .dockerignore; moved integration test data into the driver; fixed cast in oracle db_time target objective. 2019-12-04 17:27:05 -05:00
yangdsh 4e5db94697 move restart location; auto fix knob range 2019-12-03 21:28:13 -05:00
dvanaken c739eb066e Changes: removed docker centos image; updated expected value used in test in analysis/tests/ddpg_test.py to be less strict; run all tests on travis vm and docker containers; push docker images to docker-hub repo if on the master branch; 2019-12-03 18:33:04 -05:00
arifiorino cd36ff9803 Fixed Formatting 2019-12-03 17:28:35 -05:00
arifiorino 6ab7b8ef4f Bug fixes 2019-12-03 17:28:35 -05:00
yangdsh 13735c9b0b fix bugs 2019-12-03 17:28:35 -05:00
arifiorino 0dff040012 Initial code to send/manage DB error 2019-12-03 17:28:35 -05:00
dvanaken e80639c859 Fixed formatting 2019-11-26 15:25:06 -05:00
dvanaken 9994a22f16 Updated management commands 2019-11-26 15:25:06 -05:00
dvanaken 4ab2fdfd52 Updated knob tunability getter 2019-11-26 15:25:06 -05:00
dvanaken b211ad455d Fixed bug caused by stale value of latest result in views.give_result() 2019-11-26 05:15:44 -05:00
yangdsh e47508d20b change knob catelog when changing session knob tunability 2019-11-25 17:55:15 -05:00
yangdsh 73cd8489f7 fix variable name 2019-11-25 13:22:54 -05:00
yangdsh b149684d77 add GPFLOW to website 2019-11-25 13:22:54 -05:00
yangdsh dceee6e0ba support Oracle flash back; auto-fix metric type; compile ojdbc 2019-11-25 13:17:53 -05:00
dvanaken 5555ead3a3 Fixed small bug in parser. 2019-11-22 15:12:54 -05:00
dvanaken 9055516737 Updating Oracle knobs/metrics 2019-11-18 13:04:14 -05:00
dvanaken a99cf72c6c Updated oracle tunable knobs 2019-11-18 13:04:14 -05:00
dvanaken a5cbb50521 fixed formatting 2019-11-18 13:04:14 -05:00
dvanaken cf19380107 Updated the knob/metric output samples for each of the Oracle versions we support since my fix changed all of them. 2019-11-18 13:04:14 -05:00
dvanaken e1b29eaafa Added support for Oracle v12.1c. 2019-11-18 13:04:14 -05:00
dvanaken d874ca6a8b The Oracle collector now prepends the view name to each metric to avoid overwriting metrics from different views with the same names 2019-11-18 13:04:14 -05:00
dvanaken 6283186d76 Driver now works when the database system is local, remote, or on docker 2019-11-18 13:04:14 -05:00
yangdsh 21fce27291 improve ddpg 2019-11-18 10:51:22 -05:00
dvanaken 67a4a70c09 small fix 2019-11-08 05:49:19 -05:00
dvanaken abbc589880 removed invalid field 2019-11-08 05:49:19 -05:00
dvanaken 4171a662f5 Added backdoor functions for creating/editing/deleting users/projects/sessions in the website and driver. 2019-11-08 05:49:19 -05:00
dvanaken b261cf2139 Added migration for LHS 2019-11-06 21:05:33 -05:00
yangdsh e86fce59d9 import Dana's new gpr model 2019-11-04 03:15:33 -05:00
bohanjason 8bb3de85e7 update integration tests 2019-11-03 15:10:32 -05:00
bohanjason 576bbb69bf fix style 2019-11-03 15:10:32 -05:00
bohanjason 2fa97149d5 add integration test 2019-11-03 15:10:32 -05:00
yangdsh 9f71d1c8de improve simulator and ddpg 2019-10-28 20:55:06 -04:00
dvanaken 5c855faa3e Testing gitlint 2019-10-24 00:54:38 -04:00
dvanaken 9f3fe29c27 added the compression option to the main settings file 2019-10-23 22:41:55 -04:00
dvanaken 40024b4d3c Added an option to enable compression on MySQL tables (disabled by default) 2019-10-23 22:41:55 -04:00
dvanaken ec896d01f6 Updated MySQL-specific migration that enables compression to be non-atomic 2019-10-23 22:41:55 -04:00
dependabot[bot] abe439a8e1 Bump django from 1.10.1 to 1.11.23 in /server/website
Bumps [django](https://github.com/django/django) from 1.10.1 to 1.11.23.
- [Release notes](https://github.com/django/django/releases)
- [Commits](https://github.com/django/django/compare/1.10.1...1.11.23)

Signed-off-by: dependabot[bot] <support@github.com>
2019-10-23 22:41:55 -04:00
yangdsh 21f4f40b88 simplify ddpg 2019-10-23 19:54:21 -04:00
dependabot[bot] 336221d886 Bump requests from 2.18.4 to 2.20.0 in /server/website
Bumps [requests](https://github.com/requests/requests) from 2.18.4 to 2.20.0.
- [Release notes](https://github.com/requests/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/master/HISTORY.md)
- [Commits](https://github.com/requests/requests/compare/v2.18.4...v2.20.0)

Signed-off-by: dependabot[bot] <support@github.com>
2019-10-22 21:40:52 -04:00
dvanaken 777c42c979 Bumped numpy from 1.13.1 to 1.14.0 to satisfy TF 1.12.2 dependency. Updated grep commands in validator to also be compatible with BSD. 2019-10-22 19:45:39 -04:00
dependabot[bot] 68fedf765d Bump tensorflow from 1.10 to 1.12.2 in /server/website
Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 1.10 to 1.12.2.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v1.10.0...v1.12.2)

Signed-off-by: dependabot[bot] <support@github.com>
2019-10-22 19:45:39 -04:00
arifiorino b215b156a4 Moved LHS to Server 2019-10-21 17:35:28 -04:00
yangdsh a2234d08cc set default minval of knob according to hardware 2019-10-15 21:01:53 -04:00
yangdsh e4a97d060d add command resetwebsite 2019-10-15 15:01:55 -04:00
yangdsh fb7ee5c6a3 dump and load session knob tunability 2019-10-15 15:01:55 -04:00
Dana Van Aken dcb3152ec5 Added valid target objectives to the session create/edit form 2019-10-14 19:08:43 -04:00
Dana Van Aken 01b4ec3f53 Support adding custom target objectives to the website/db/*/target_objective.py modules 2019-10-14 19:08:43 -04:00
Dana Van Aken f68c23e975 Added outer db module and moved parser code into it 2019-10-14 19:08:43 -04:00
yangdsh a3833d83b1 normalize reward; square reward 2019-10-14 17:52:04 -04:00
yangdsh 090387a176 change the expected value in ddpg test 2019-10-14 17:52:04 -04:00
yangdsh f071a0e62c modify the hyper parameter and reward of ddpg 2019-10-14 17:52:04 -04:00
Dana Van Aken 0da1b724cc Added postgresql client package to Dockerfiles and psycopg2-binary to requirements.txt, removed unused matplotlib code/package from analysis/cluster.py and requirements.txt, replaced commands in docker start.sh file with new management commands. 2019-10-08 09:32:25 -04:00
Dana Van Aken e8f28ebef0 Added another function that converts bytes/ms to human-readable values without loss of resolution. 2019-10-08 09:32:25 -04:00
Dana Van Aken 3d70990ff1 The calculation of maxval for session knobs that use memory/storage resources is now set to 80% of the available resource (versus 100%) 2019-10-08 09:32:25 -04:00
Dana Van Aken c568c09c00 Enforce unique Project.name for same user and unique Session.name for same user and project 2019-10-08 09:32:25 -04:00
Dana Van Aken 40c75de3ce Added additional Oracle views to metric collection (v, v). 2019-10-08 09:32:25 -04:00
Dana Van Aken 162dc48c53 Added option to dump debug info in 3 places: (1) the session view, (2) as a command in manage.py, and (3) as a fab command in the driver. 2019-10-08 09:32:25 -04:00
DongshengYang f08204b9d6 Add testing for ddpg model 2019-10-08 08:58:51 -04:00
Bohan Zhang 876d975713 improve UI 2019-10-08 08:29:10 -04:00
bohanjason c5702e9159 tweak alignment in UI 2019-10-08 08:29:10 -04:00
bohanjason 924cccc459 update navbar in UI 2019-10-08 08:29:10 -04:00
bohanjason 63b7dd9a77 modify session page UI 2019-10-08 08:29:10 -04:00
Dana Van Aken e676433ece Changes:
- fixed bug in validator causing it to validate files in excluded directories and updated the pre-commit hook
 - hardware uniqueness is now based on the cpu, memory, storage, and storage_type fields, fixes #230
 - initial settings of a minval/maxval for a session knob are calculated based on its resource type and the user's hardware
 - squashed migrations
2019-10-03 21:31:27 -04:00
Dana Van Aken c14a337695 Removed obsolete methods from parser and fields from Session model 2019-10-03 21:31:27 -04:00
Dana Van Aken b3c42a81fb Added django-db-logger for storing log messages in the database, removed random *.sh scripts from website code then extended manage.py with same commands 2019-10-03 21:31:27 -04:00
Dana Van Aken 6b8aeb6043 fixed expected status_codes in tests 2019-10-03 21:31:27 -04:00
Dana Van Aken 390e58b24d added some manage.py extensions 2019-10-03 21:31:27 -04:00
Dana Van Aken e5edd2a7f4 Fixed celery task logging and added filehandler 2019-10-03 21:31:27 -04:00
Dana Van Aken 863d6619ba Fixed incorrect parsing behavior for float-type metrics 2019-10-03 21:31:27 -04:00
Dana Van Aken d1c13de42f Consolidated driver scripts and added status codes to the query_and_get and new_result responses 2019-10-03 21:31:27 -04:00
DongshengYang 8743c0c2b1 support oracle v12 in ottertune server 2019-10-03 18:58:30 -04:00
bohanjason 494b23678e use algorithm tpye 2019-09-30 01:34:30 -04:00
bohanjason e9f503ef3e change to algoroithm tpyes 2019-09-30 01:34:30 -04:00
bohanjason 25d0838376 save dnn model into database 2019-09-30 01:34:30 -04:00
bohanjason c37ef9c072 move dnn parameters to constant file 2019-09-30 01:34:30 -04:00
bohanjason 82a7d859c2 choose algorithm based on option 2019-09-30 01:34:30 -04:00
bohanjason e1b97bca9b upgrade tensorflow 2019-09-30 01:34:30 -04:00
bohanjason f716ade292 add noise exploration 2019-09-30 01:34:30 -04:00
bohanjason 11cfe5afc1 add deep learning model 2019-09-30 01:34:30 -04:00