remove keys (#225)

* new fetch_calendar_data function

* new class Calendar

* Revert common/model.py to the state before the last commit

* dataframe transformation is making Timestamp objects in open and close columns naive before converting to dictionary.

* typing for function arguments added. Function returns list of Calendar objects that have properties defined like strings.

* else condition fixed

* else condition returns directly an  empty without declaring a list name

README.md

@@ -1,9 +1,9 @@
-**README - V2TRADING - Advanced Algorithmic Trading Platform**
+# V2TRADING - Advanced Algorithmic Trading Platform
 
-**Overview**
+## Overview
-Custom-built algorithmic trading platform for research, backtesting and automated trading. Trading engine capable of processing tick data, managing trades, and supporting backtesting in a highly accurate and efficient manner.
+Custom-built algorithmic trading platform for research, backtesting and live trading. Trading engine capable of processing tick data, providing custom aggregation, managing trades, and supporting backtesting in a highly accurate and efficient manner.
 
-**Key Features**
+## Key Features
 - **Trading Engine**: At the core of the platform is a trading engine that processes tick data in real time. This engine is responsible for aggregating data and managing the execution of trades, ensuring precision and speed in trade placement and execution.
 
 - **High-Fidelity Backtesting Environment**: ability to backtest strategies with 1:1 precision - meaning a tick-by-tick backtesting. This level of precision in backtesting, down to millisecond accuracy, mirrors live trading environments and is vital for developing and testing high-frequency trading strategies.
@@ -51,3 +51,78 @@ This repository represents a sophisticated and evolving tool for algorithmic tra
 </p>
 
 
+# Installation Instructions
+This document outlines the steps for installing and setting up the necessary environment for the application. These instructions are applicable for both Windows and Linux operating systems. Please follow the steps carefully to ensure a smooth setup.
+
+## Prerequisites
+Before beginning the installation process, ensure the following prerequisites are met:
+
+- TA-Lib Library:
+  - Windows: Download and build the TA-Lib library. Install Visual Studio Community with the Visual C++ feature. Navigate to `C:\ta-lib\c\make\cdr\win32\msvc` in the command prompt and build the library using the available makefile.
+  - Linux: Install TA-Lib using your distribution's package manager or compile from source following the instructions available on the TA-Lib GitHub repository.
+ 
+- Alpaca Paper Trading Account: Create an account at [Alpaca Markets](https://alpaca.markets/) and generate `API_KEY` and `SECRET_KEY` for your paper trading account.
+
+## Installation Steps
+**Clone the Repository:** Clone the remote repository to your local machine.
+   `git clone git@github.com:drew2323/v2trading.git <name_of_local_folder>`
+   
+**Install Python:** Ensure Python 3.10.11 is installed on your system.
+
+**Create a Virtual Environment:** Set up a Python virtual environment.
+   `python -m venv <path_to_venv_folder>`
+   
+**Activate Virtual Environment:**
+   - Windows: `source ./<venv_folder>/Scripts/activate`
+   - Linux: `source ./<venv_folder>/bin/activate`
+
+**Install Dependencies:** Install the program requirements.
+   pip install -r requirements.txt
+   Note: It's permissible to comment out references to `keras` and `tensorflow` modules, as well as the `ml-room` repository in `requirements.txt`.
+
+**Environment Variables:** In `run.sh`, modify the `VIRTUAL_ENV_DIR` and `PYTHON_TO_USE` variables as necessary.
+
+**Data Directory:** Navigate to `DATA_DIR` and create folders: `aggcache`, `tradecache`, and `models`.
+
+**Media and Static Folders:** Create `media` and `static` folders one level above the repository directory. Also create `.env` file there.
+
+**Database Setup:** Create the `v2trading.db` file using SQL commands from `v2trading_create_db.sql`.
+```
+    import sqlite3
+    with open("v2trading_create_db.sql", "r") as f:
+        sql_statements = f.read()
+    conn = sqlite3.connect('v2trading.db')
+    cursor = conn.cursor()
+    cursor.executescript(sql_statements)
+    conn.commit()
+    conn.close()
+```
+Ensure the `config_table` is not empty by making an initial entry.
+
+**Start the Application:** Run `main.py` in VSCode to start the application.
+
+**Accessing the Application:** If the uvicorn server runs successfully at `http://0.0.0.0:8000`, access the application at `http://localhost:8000/static/`.
+
+**Database Configuration:** Add dynamic button and JS configurations to the `config_table` in `v2trading.db` via the "Config" section on the main page.
+Please replace placeholders (e.g., `<name_of_local_folder>`, `<path_to_venv_folder>`) with your actual paths and details. Follow these instructions to ensure the application is set up correctly and ready for use.
+
+## Environmental variables
+Trading platform can support N different accounts. Their API keys are stored as environmental variables in .env file located in the root directory.
+Account for trading api is selected when each strategy is run. However for realtime websocket data), always ACCOUNT1 is used for all strategies. The data point selection (iex vs sip) is set by LIVE_DATA_FEED environment variable.  
+
+.env file should contain:
+
+```
+ACCOUNT1_LIVE_API_KEY=<ACCOUNT1_LIVE_API_KEY>
+ACCOUNT1_LIVE_SECRET_KEY=<ACCOUNT1_LIVE_SECRET_KEY>
+ACCOUNT1_LIVE_FEED=sip
+ACCOUNT1_PAPER_API_KEY=<ACCOUNT1_PAPER_API_KEY>
+ACCOUNT1_PAPER_SECRET_KEY=<ACCOUNT1_PAPER_SECRET_KEY>
+ACCOUNT1_PAPER_FEED=sip
+ACCOUNT2_PAPER_API_KEY=<ACCOUNT2_PAPER_API_KEY>
+ACCOUNT2_PAPER_SECRET_KEY=ACCOUNT2_PAPER_SECRET_KEY<>
+ACCOUNT2_PAPER_FEED=iex
+WEB_API_KEY=<pass-for-webapi>
+```
+
+

requirements_new.txt

@@ -236,7 +236,7 @@ tzlocal==4.3
 uri-template==1.3.0
 urllib3==1.26.14
 uvicorn==0.21.1
--e git+https://github.com/drew2323/v2trading.git@78f2162d59753c243e374dd52c305e111affebb9#egg=v2realbot
+-e git+https://github.com/drew2323/v2trading.git@1f85b271dba2b9baf2c61b591a08849e9d684374#egg=v2realbot
 validators==0.20.0
 vectorbtpro @ file:///Users/davidbrazda/Downloads/vectorbt.pro-2024.2.22
 wcwidth==0.2.9

requirements_newest.txt

@@ -0,0 +1,243 @@
+absl-py
+alpaca
+alpaca-py
+altair
+annotated-types
+anyio
+appdirs
+appnope
+APScheduler
+argon2-cffi
+argon2-cffi-bindings
+arrow
+asttokens
+astunparse
+async-lru
+attrs
+Babel
+beautifulsoup4
+better-exceptions
+bleach
+blinker
+bottle
+cachetools
+CD
+certifi
+cffi
+chardet
+charset-normalizer
+click
+colorama
+comm
+contourpy
+cycler
+dash
+dash-bootstrap-components
+dash-core-components
+dash-html-components
+dash-table
+dateparser
+debugpy
+decorator
+defusedxml
+dill
+dm-tree
+entrypoints
+exceptiongroup
+executing
+fastapi
+fastjsonschema
+filelock
+Flask
+flatbuffers
+fonttools
+fpdf2
+fqdn
+gast
+gitdb
+GitPython
+google-auth
+google-auth-oauthlib
+google-pasta
+greenlet
+grpcio
+h11
+h5py
+html2text
+httpcore
+httpx
+humanize
+icecream
+idna
+imageio
+importlib-metadata
+ipykernel
+ipython
+ipywidgets
+isoduration
+itables
+itsdangerous
+jax
+jaxlib
+jedi
+Jinja2
+joblib
+json5
+jsonpointer
+jsonschema
+jsonschema-specifications
+jupyter-events
+jupyter-lsp
+jupyter_client
+jupyter_core
+jupyter_server
+jupyter_server_terminals
+jupyterlab
+jupyterlab-widgets
+jupyterlab_pygments
+jupyterlab_server
+kaleido
+keras
+keras-core
+keras-nightly
+keras-nlp-nightly
+keras-tcn @ git+https://github.com/drew2323/keras-tcn.git
+kiwisolver
+libclang
+lightweight-charts @ git+https://github.com/drew2323/lightweight-charts-python.git
+llvmlite
+Markdown
+markdown-it-py
+MarkupSafe
+matplotlib
+matplotlib-inline
+mdurl
+mistune
+ml-dtypes
+mlroom @ git+https://github.com/drew2323/mlroom.git
+mplfinance
+msgpack
+mypy-extensions
+namex
+nbclient
+nbconvert
+nbformat
+nest-asyncio
+newtulipy
+notebook_shim
+numba
+numpy
+oauthlib
+opt-einsum
+orjson
+overrides
+packaging
+pandas
+pandocfilters
+param
+parso
+patsy
+pexpect
+Pillow
+platformdirs
+plotly
+prometheus_client
+prompt-toolkit
+proto-plus
+protobuf
+proxy-tools
+psutil
+ptyprocess
+pure-eval
+pyarrow
+pyasn1
+pyasn1-modules
+pycparser
+pyct
+pydantic
+pydantic_core
+pydeck
+Pygments
+pyinstrument
+pyparsing
+pyrsistent
+pysos
+python-dateutil
+python-dotenv
+python-json-logger
+python-multipart
+pytz
+pytz-deprecation-shim
+pyviz-comms
+PyWavelets
+pywebview
+PyYAML
+pyzmq
+referencing
+regex
+requests
+requests-oauthlib
+rfc3339-validator
+rfc3986-validator
+rich
+rpds-py
+rsa
+schedule
+scikit-learn
+scipy
+seaborn
+semver
+Send2Trash
+six
+smmap
+sniffio
+soupsieve
+SQLAlchemy
+sseclient-py
+stack-data
+starlette
+statsmodels
+streamlit
+structlog
+TA-Lib
+tb-nightly
+tenacity
+tensorboard
+tensorboard-data-server
+tensorflow-addons
+tensorflow-estimator
+tensorflow-io-gcs-filesystem
+termcolor
+terminado
+tf-estimator-nightly
+tf-nightly
+tf_keras-nightly
+threadpoolctl
+tinycss2
+tinydb
+tinydb-serialization
+tinyflux
+toml
+tomli
+toolz
+tornado
+tqdm
+traitlets
+typeguard
+types-python-dateutil
+typing_extensions
+tzdata
+tzlocal
+uri-template
+urllib3
+uvicorn
+validators
+wcwidth
+webcolors
+webencodings
+websocket-client
+websockets
+Werkzeug
+widgetsnbextension
+wrapt
+zipp

research/get_trades_at_once.ipynb

@@ -402,7 +402,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.10"
   }
  },
  "nbformat": 4,

research/strat_LINREG_MULTI/v1_SINGLE.ipynb

@@ -44661,7 +44661,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.10"
   }
  },
  "nbformat": 4,

research/strat_ORDER_IMBALANCE/v2_SINGLE.ipynb

@@ -211,13 +211,6 @@
     "t1data.data[\"BAC\"].info()"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
   {
    "cell_type": "code",
    "execution_count": 20,
@@ -1698,7 +1691,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.10"
   }
  },
  "nbformat": 4,

testdoc.md

@@ -0,0 +1,107 @@
+# Plotly
+
+* MAKE_SUBPLOT Defines layout (if more then 1x1 or secondary y axis are required)
+
+```python
+fig = vbt.make_subplots(rows=2, cols=1, shared_xaxes=True, 
+                        specs=[[{"secondary_y": True}], [{"secondary_y": False}]], 
+                        vertical_spacing=0.02, subplot_titles=("Row 1 title", "Row 2 title"))
+```
+
+Then the different [sr/df generic accessor](http://5.161.179.223:8000/static/js/vbt/api/generic/accessors/index.html#vectorbtpro.generic.accessors.GenericAccessor.areaplot) are added with ADD_TRACE_KWARGS and TRACE_KWARGS. Other types of plot available in [plotting module](http://5.161.179.223:8000/static/js/vbt/api/generic/plotting/index.html)
+
+```python
+#using accessor
+close.vbt.plot(fig=fig, add_trace_kwargs=dict(secondary_y=False,row=1, col=1), trace_kwargs=dict(line=dict(color="blue")))
+indvolume.vbt.barplot(fig=fig, add_trace_kwargs=dict(secondary_y=False, row=2, col=1))
+#using plotting module
+vbt.Bar(indvolume, fig=fig, add_trace_kwargs=dict(secondary_y=False, row=2, col=1))
+```
+
+* ADD_TRACE_KWARGS - determines positioning withing subplot
+```python
+add_trace_kwargs=dict(secondary_y=False,row=1, col=1)
+```
+* TRACE_KWARGS - other styling of trace
+```python
+trace_kwargs=dict(name="LONGS",
+                  line=dict(color="#ffe476"),
+                  marker=dict(color="limegreen"),
+                  fill=None,
+                  connectgaps=True)
+```
+
+## Example
+
+```python
+    fig = vbt.make_subplots(rows=2, cols=1, shared_xaxes=True, 
+                            specs=[[{"secondary_y": True}], [{"secondary_y": False}]], 
+                            vertical_spacing=0.02, subplot_titles=("Price and Indicators", "Volume"))
+
+    # Plotting the close price
+    close.vbt.plot(fig=fig, add_trace_kwargs=dict(secondary_y=False,row=1, col=1), trace_kwargs=dict(line=dict(color="blue")))
+```
+
+# Data
+## Resampling
+```python
+t1data = basic_data[['open', 'high', 'low', 'close', 'volume','vwap','buyvolume','sellvolume']].resample("1T")
+t1data = t1data.transform(lambda df: df.between_time('09:30', '16:00').dropna()) #main session data only, no nans
+
+t5data = basic_data[['open', 'high', 'low', 'close', 'volume','vwap','buyvolume','sellvolume']].resample("5T")
+t5data = t5data.transform(lambda df: df.between_time('09:30', '16:00').dropna())
+
+dailydata = basic_data[['open', 'high', 'low', 'close', 'volume', 'vwap']].resample("D").dropna()
+
+#realign 5min close to 1min so it can be compared with 1min
+t5data_close_realigned = t5data.close.vbt.realign_closing("1T").between_time('09:30', '16:00').dropna() 
+#same with open
+t5data.open.vbt.realign_opening("1h")
+```
+### Define resample function for custom column
+Example of custom feature config [Binance Data](http://5.161.179.223:8000/static/js/vbt/api/data/custom/binance/index.html#vectorbtpro.data.custom.binance.BinanceData.feature_config).
+Other [reduced functions available](http://5.161.179.223:8000/static/js/vbt/api/generic/nb/apply_reduce/index.html). (mean, min, max, median, nth ...)
+```python
+from vectorbtpro.utils.config import merge_dicts, Config, HybridConfig
+from vectorbtpro import _typing as tp
+from vectorbtpro.generic import nb as generic_nb
+
+_feature_config: tp.ClassVar[Config] = HybridConfig(
+    {
+        "buyvolume": dict(
+            resample_func=lambda self, obj, resampler: obj.vbt.resample_apply(
+                resampler,
+                generic_nb.sum_reduce_nb,
+            )
+        ),
+        "sellvolume": dict(
+            resample_func=lambda self, obj, resampler: obj.vbt.resample_apply(
+                resampler,
+                generic_nb.sum_reduce_nb,
+            )
+        )
+    }
+)
+
+basic_data._feature_config = _feature_config
+```
+
+### Validate resample
+```python
+t2dataclose = t2data.close.rename("15MIN - realigned").vbt.realign_closing("1T")
+fig = t1data.close.rename("1MIN").vbt.plot()
+t2data.close.rename("15MIN").vbt.plot(fig=fig)
+t2dataclose.vbt.plot(fig=fig)
+```
+## Persisting
+```python
+basic_data.to_parquet(partition_by="day", compression="gzip")  
+day_data = vbt.ParquetData.pull("BAC", filters=[("group", "==", "2024-05-03")]) 
+vbt.print_dir_tree("BTC-USD")#overeni directory structure
+```
+# Discover
+```python
+vbt.phelp(vbt.talib(“atr”).run) #parameters it accepts
+vbt.pdir(pf) - get available properties and methods
+vbt.pprint(basic_data) #to get correct shape, info about instance
+```

testy/archive/config.py

@@ -1,3 +1,3 @@
-API_KEY = 'PKGGEWIEYZOVQFDRY70L'
+API_KEY = ''
-SECRET_KEY = 'O5Kt8X4RLceIOvM98i5LdbalItsX7hVZlbPYHy8Y'
+SECRET_KEY = ''
 MAX_BATCH_SIZE = 1

v2realbot/main.py

@@ -18,6 +18,7 @@ from fastapi.security import HTTPBasic, HTTPBasicCredentials
 from v2realbot.enums.enums import Env, Mode
 from typing import Annotated
 import os
+import psutil
 import uvicorn
 import orjson
 from queue import Queue, Empty
@@ -1025,7 +1026,25 @@ def get_metadata(model_name: str):
     #         "last_modified": os.path.getmtime(model_path),
     #         # ... other metadata fields ...
     #     }
-
+@app.get("/system-info")
+def get_system_info():
+    """Get system info, e.g. disk free space, used percentage ... """
+    disk_total = round(psutil.disk_usage('/').total / 1024**3, 1)
+    disk_used = round(psutil.disk_usage('/').used / 1024**3, 1)
+    disk_free = round(psutil.disk_usage('/').free / 1024**3, 1)
+    disk_used_percentage = round(psutil.disk_usage('/').percent, 1)
+    # memory_total = round(psutil.virtual_memory().total / 1024**3, 1)
+    # memory_perc = round(psutil.virtual_memory().percent, 1)
+    # cpu_time_user = round(psutil.cpu_times().user,1)
+    # cpu_time_system = round(psutil.cpu_times().system,1)
+    # cpu_time_idle = round(psutil.cpu_times().idle,1)
+    # network_sent = round(psutil.net_io_counters().bytes_sent / 1024**3, 6)
+    # network_recv = round(psutil.net_io_counters().bytes_recv / 1024**3, 6)
+    return {"disk_space": {"total": disk_total, "used": disk_used, "free" : disk_free, "used_percentage" : disk_used_percentage},
+            # "memory": {"total": memory_total, "used_percentage": memory_perc},
+            # "cpu_time" : {"user": cpu_time_user, "system": cpu_time_system, "idle": cpu_time_idle},
+            # "network": {"sent": network_sent, "received": network_recv}
+            }
 
 # Thread function to insert data from the queue into the database
 def insert_queue2db():

v2realbot/static/index.html

@@ -131,9 +131,29 @@
 
         <!-- <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/monaco-editor/0.41.0/min/vs/editor/editor.main.js"></script>  -->
         <!-- <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/monaco-editor/0.41.0/min/vs/loader.min.js"></script>   -->
+        <script src="/static/js/systeminfo.js"> </script>
     </head>
     <body>
         <div id="main" class="mainConteiner flex-container content">
+            <div id="system-info" class="flex-items"> 
+                <label data-bs-toggle="collapse" data-bs-target="#system-info-inner" aria-expanded="true">
+                    <h4>System Info </h4>
+                </label>
+                <div id="system-info-inner" class="collapse">
+                    <div id="system-info-output"></div>
+                    <div id="graphical-output"> 
+                        <div id="disk-gauge-container">
+                            <span id="title"> Disk Space: </span>
+                            <span id="free-space">Free: -- GB</span> | 
+                            <span id="total-space">Total: -- GB</span> | 
+                            <span id="used-percent">Used: -- %</span>
+                            <div id="disk-gauge">
+                            <div id="disk-gauge-bar"></div>
+                            </div>
+                        </div>
+                    </div>
+                </div>
+            </div>
             <div id="chartContainer" class="flex-items">
                 <label data-bs-toggle="collapse" data-bs-target="#chartContainerInner" aria-expanded="true">
                     <h4>Chart</h4>
@@ -230,6 +250,7 @@
                         <!-- <table id="trades-data-table" class="dataTable no-footer" style="width: 300px;display: contents;"></table> -->
                 </div>
             </div>
+
             <div id="runner-table" class="flex-items">
                 <label data-bs-toggle="collapse" data-bs-target="#runner-table-inner">
                     <h4>Running Strategies</h4>

v2realbot/static/js/systeminfo.js

@@ -0,0 +1,31 @@
+    function get_system_info() {
+        console.log('Button get system status clicked')
+        $.ajax({
+            url: '/system-info',
+            type: 'GET',
+            beforeSend: function (xhr) {
+                xhr.setRequestHeader('X-API-Key',
+                API_KEY); },
+            success: function(response) {
+                $.each(response, function(index, item) {
+                    if (index=="disk_space") {
+                        $('#disk-gauge-bar').css('width', response.disk_space.used_percentage + '%');
+                        $('#free-space').text('Free: ' + response.disk_space.free + ' GB');
+                        $('#total-space').text('Total: ' + response.disk_space.total + ' GB');
+                        $('#used-percent').text('Used: ' + response.disk_space.used_percentage + '%');
+                    } else {
+                        var formatted_item = JSON.stringify(item, null, 4)
+                        $('#system-info-output').append('<p>' + index + ': ' + formatted_item + '</p>');
+                    }
+                });            
+            },
+            error: function(xhr, status, error) {
+                $('#disk-gauge-bar').html('An error occurred: ' + error + xhr.responseText + status);
+            }
+        });
+    }
+
+
+$(document).ready(function(){
+  get_system_info()
+});