Compare commits

...

5 Commits

Author SHA1 Message Date
14ad709b53 gitgnore 2025-12-08 12:51:50 +01:00
ccfb40a2b3 cleanup add training bell 2025-12-08 12:26:34 +01:00
Philipp
036f3b178a que and training status 2025-12-02 12:50:22 +01:00
Philipp
5bfe82fc26 addet training info 2025-12-02 10:51:20 +01:00
Philipp
de5a8d2028 add multi trainings +fix path for coco retraining 2025-12-02 10:16:18 +01:00
3086 changed files with 671461 additions and 70218 deletions

2
.gitignore vendored Normal file → Executable file
View File

@@ -3,3 +3,5 @@ backend/uploads/*.pth
*.pth *.pth
backend/node_modules/ backend/node_modules/
backend/.venv/ backend/.venv/
key
key.pub

View File

@@ -1,217 +0,0 @@
# Transfer Learning Base Configuration Feature
## Overview
This feature implements automatic loading of base configurations when "Train on COCO" transfer learning is selected. Base parameters are loaded from `backend/data/` based on the selected YOLOX model, and these protected fields are displayed as greyed out and non-editable in the frontend.
## Components Modified/Created
### Backend
#### 1. Base Configuration Files (`backend/data/`)
- **`yolox_s.py`** - Base config for YOLOX-Small (depth=0.33, width=0.50)
- **`yolox_m.py`** - Base config for YOLOX-Medium (depth=0.67, width=0.75)
- **`yolox_l.py`** - Base config for YOLOX-Large (depth=1.0, width=1.0)
- **`yolox_x.py`** - Base config for YOLOX-XLarge (depth=1.33, width=1.25)
Each file contains a `BaseExp` class with protected parameters:
- Model architecture (depth, width, activation)
- Training hyperparameters (max_epoch, warmup_epochs, scheduler, etc.)
- Optimizer settings (momentum, weight_decay)
- Augmentation probabilities (mosaic_prob, mixup_prob, etc.)
- Input/output sizes
#### 2. Services (`backend/services/generate_yolox_exp.py`)
**New functions:**
- `load_base_config(selected_model)` - Dynamically loads base config using importlib
- Modified `generate_yolox_inference_exp()` to support `use_base_config` parameter
- Base config merging logic: base → user overrides → defaults
**Behavior:**
- `transfer_learning='coco'` → loads base config + applies user overrides
- `transfer_learning='sketch'` → uses only user-defined values
- Protected parameters from base config are preserved unless explicitly overridden
#### 3. API Routes (`backend/routes/api.py`)
**New endpoint:**
```python
@api_bp.route('/base-config/<model_name>', methods=['GET'])
def get_base_config(model_name):
```
Returns the base configuration JSON for a specific YOLOX model.
### Frontend
#### 1. HTML (`edit-training.html`)
**Added:**
- Info banner to indicate when base config is active
- CSS styles for disabled input fields (grey background, not-allowed cursor)
- Visual feedback showing which model's base config is loaded
**Banner HTML:**
```html
<div id="base-config-info" style="display:none; ...">
🔒 Base Configuration Active
Protected parameters are loaded from [model] base config
</div>
```
**CSS for disabled fields:**
```css
.setting-row input[type="number"]:disabled,
.setting-row input[type="text"]:disabled,
.setting-row input[type="checkbox"]:disabled {
background: #d3d3d3 !important;
color: #666 !important;
cursor: not-allowed !important;
border: 1px solid #999 !important;
}
```
#### 2. JavaScript (`js/start-training.js`)
**New functionality:**
1. **Base Config Loading:**
```javascript
function loadBaseConfig(modelName)
```
Fetches base config from `/api/base-config/<model>`
2. **Apply Base Config:**
```javascript
function applyBaseConfig(config, isCocoMode)
```
- Applies config values to form fields
- Disables and greys out protected fields
- Shows/hides info banner
- Adds tooltips to disabled fields
3. **Update Transfer Learning Mode:**
```javascript
function updateTransferLearningMode()
```
- Monitors changes to "Transfer Learning" dropdown
- Monitors changes to "Select Model" dropdown
- Loads appropriate base config when COCO mode is selected
- Clears base config when sketch mode is selected
4. **Form Submission Enhancement:**
- Temporarily enables disabled fields before submission
- Ensures protected parameters are included in form data
- Re-disables fields after collection
**Protected Fields List:**
```javascript
const protectedFields = [
'depth', 'width', 'act', 'max_epoch', 'warmup_epochs', 'warmup_lr',
'scheduler', 'no_aug_epochs', 'min_lr_ratio', 'ema', 'weight_decay',
'momentum', 'input_size', 'mosaic_scale', 'test_size', 'enable_mixup',
'mosaic_prob', 'mixup_prob', 'hsv_prob', 'flip_prob', 'degrees',
'translate', 'shear', 'mixup_scale', 'print_interval', 'eval_interval'
];
```
## User Flow
### 1. Normal Custom Training (Train from sketch)
- User selects model: e.g., "YOLOX-s"
- User selects "Train from sketch"
- All fields are editable (white background)
- User can customize all parameters
- Submission uses user-defined values only
### 2. COCO Transfer Learning (Train on COCO)
- User selects model: e.g., "YOLOX-s"
- User selects "Train on coco"
- **Automatic actions:**
1. Frontend calls `/api/base-config/YOLOX-s`
2. Base config is loaded and applied
3. Protected fields become greyed out and disabled
4. Green info banner appears: "🔒 Base Configuration Active"
5. Tooltip on hover: "Protected by base config for YOLOX-s. Switch to 'Train from sketch' to customize."
- User can still edit non-protected fields
- On submit: both base config values AND user overrides are sent to backend
- Backend generates exp.py with merged settings
### 3. Switching Models
- User changes from "YOLOX-s" to "YOLOX-l" (while in COCO mode)
- Frontend automatically:
1. Fetches new base config for YOLOX-l
2. Updates field values (depth=1.0, width=1.0, etc.)
3. Updates banner text to show "YOLOX-l"
- Protected parameters update to match new model's architecture
## Testing
### Manual Test Steps:
1. **Test Base Config Loading:**
```bash
cd backend/data
python test_base_configs.py
```
Should display all parameters for yolox-s, yolox-m, yolox-l, yolox-x
2. **Test API Endpoint:**
```bash
# Start Flask server
cd backend
python app.py
# In another terminal:
curl http://localhost:3000/api/base-config/YOLOX-s
```
Should return JSON with depth, width, activation, etc.
3. **Test Frontend:**
- Open `edit-training.html?id=<project_id>` in browser
- Select "YOLOX-s" model
- Select "Train on coco" → fields should grey out
- Select "Train from sketch" → fields should become editable
- Switch to "YOLOX-l" (in COCO mode) → values should update
- Open browser console and check for: `Applied base config. Protected fields: depth, width, ...`
4. **Test Form Submission:**
- With COCO mode active (fields greyed out)
- Click "Save Parameters"
- Check browser Network tab → POST to `/api/yolox-settings`
- Verify payload includes protected parameters (depth, width, etc.)
- Check Flask logs for successful save
### Expected Behaviors:
✅ **COCO mode + YOLOX-s:**
- depth: 0.33 (greyed out)
- width: 0.50 (greyed out)
- activation: silu (greyed out)
- Info banner visible
✅ **COCO mode + YOLOX-l:**
- depth: 1.0 (greyed out)
- width: 1.0 (greyed out)
- activation: silu (greyed out)
✅ **Sketch mode:**
- All fields white/editable
- No info banner
- User can set any values
## Documentation
- **`backend/data/README.md`** - Complete guide on base config system
- **`backend/data/test_base_configs.py`** - Test script for base configs
## Benefits
1. **Proven defaults:** Users start with battle-tested COCO pretraining settings
2. **Prevents mistakes:** Can't accidentally break model architecture by changing depth/width
3. **Easy customization:** Can still override specific parameters if needed
4. **Visual feedback:** Clear indication of which fields are protected
5. **Model-specific:** Each model (s/m/l/x) has appropriate architecture defaults
6. **Flexible:** Can easily add new models by creating new base config files
## Future Enhancements
- Add "Override" button next to protected fields to unlock individual parameters
- Show diff comparison between base config and user overrides
- Add validation warnings if user tries values far from base config ranges
- Export final merged config as preview before training

124
add-project.html Normal file → Executable file
View File

@@ -29,6 +29,14 @@
<icon class="header-icon" onclick="window.location.href='/index.html'" , onmouseover="" <icon class="header-icon" onclick="window.location.href='/index.html'" , onmouseover=""
style="cursor: pointer;" src="./media/logo.png" alt="Logo"></icon> style="cursor: pointer;" src="./media/logo.png" alt="Logo"></icon>
<div class="button-row"> <div class="button-row">
<!-- Training Notification Bell -->
<button id="training-bell" onclick="toggleTrainingModal()" class="button" title="Training Status"
style="padding: 8px 16px; margin-right: 10px; position: relative; background: #999;">
🔔
<span id="bell-badge" style="display: none; position: absolute; top: -5px; right: -5px; background: #ff4d4f;
color: white; border-radius: 50%; width: 20px; height: 20px; font-size: 12px; line-height: 20px;
text-align: center; font-weight: bold;">0</span>
</button>
<button id="Add Training Project" class="button-red">Add Training Project</button> <button id="Add Training Project" class="button-red">Add Training Project</button>
<button id="Add Dataset" class="button">Add Dataset</button> <button id="Add Dataset" class="button">Add Dataset</button>
<button id="Import Dataset" class="button">Refresh Label-Studio</button> <button id="Import Dataset" class="button">Refresh Label-Studio</button>
@@ -234,7 +242,123 @@
</div> </div>
</div> </div>
<!-- Training Status Modal -->
<div id="training-status-modal" class="modal" style="display: none;">
<div class="modal-content" style="max-width: 700px; max-height: 90vh; overflow-y: auto;">
<div class="modal-header">
<h2>Training Status</h2>
<button class="close-btn" onclick="toggleTrainingModal()">&times;</button>
</div>
<div class="modal-body">
<div class="settings-section" id="current-training-section" style="display: none;">
<h3 style="color: #009eac;">Current Training</h3>
<div id="current-training-info" style="background: #eaf7fa; padding: 16px; border-radius: 8px; margin-bottom: 16px;">
</div>
</div>
<div class="settings-section" id="queue-section" style="display: none;">
<h3 style="color: #666;">Queue (<span id="queue-count">0</span>)</h3>
<div id="queue-list" style="display: flex; flex-direction: column; gap: 12px;">
</div>
</div>
<div id="no-trainings-msg" style="text-align: center; padding: 32px; color: #666;">
No trainings running or queued.
</div>
</div>
</div>
</div>
<script src="js/settings.js"></script> <script src="js/settings.js"></script>
<script>
let trainingStatusPoller = null;
function toggleTrainingModal() {
const modal = document.getElementById('training-status-modal');
if (modal.style.display === 'none') {
modal.style.display = 'flex';
updateTrainingStatus();
} else {
modal.style.display = 'none';
}
}
function updateTrainingStatus() {
fetch('/api/training-status')
.then(res => res.json())
.then(data => {
const bell = document.getElementById('training-bell');
const badge = document.getElementById('bell-badge');
const currentSection = document.getElementById('current-training-section');
const queueSection = document.getElementById('queue-section');
const noTrainingsMsg = document.getElementById('no-trainings-msg');
const totalCount = (data.current ? 1 : 0) + data.queue.length;
if (totalCount > 0) {
bell.style.background = '#009eac';
badge.style.display = 'block';
badge.textContent = totalCount;
} else {
bell.style.background = '#999';
badge.style.display = 'none';
}
if (data.current) {
currentSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
const percentage = Math.round((data.current.iteration / data.current.max_epoch) * 100);
document.getElementById('current-training-info').innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px;">
<strong>${data.current.name || 'Training'}</strong>
<span style="font-weight: bold; color: #009eac;">${percentage}%</span>
</div>
<div style="background: #ddd; border-radius: 4px; height: 24px; overflow: hidden; margin-bottom: 8px;">
<div style="background: #009eac; height: 100%; width: ${percentage}%; transition: width 0.3s;"></div>
</div>
<div style="font-size: 14px; color: #666;">
Epoch ${data.current.iteration} / ${data.current.max_epoch}
</div>
`;
} else {
currentSection.style.display = 'none';
}
if (data.queue.length > 0) {
queueSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
document.getElementById('queue-count').textContent = data.queue.length;
document.getElementById('queue-list').innerHTML = data.queue.map((t, idx) => `
<div style="background: #f5f5f5; padding: 12px; border-radius: 8px; border-left: 4px solid #009eac;">
<strong>#${idx + 1}: ${t.name || 'Training'}</strong>
<div style="font-size: 13px; color: #666; margin-top: 4px;">
${t.max_epoch} epochs • Waiting...
</div>
</div>
`).join('');
} else {
queueSection.style.display = 'none';
}
if (totalCount === 0) {
noTrainingsMsg.style.display = 'block';
}
})
.catch(err => console.error('Failed to fetch training status:', err));
}
window.addEventListener('DOMContentLoaded', function() {
updateTrainingStatus();
trainingStatusPoller = setInterval(updateTrainingStatus, 5000);
});
window.addEventListener('beforeunload', function() {
if (trainingStatusPoller) clearInterval(trainingStatusPoller);
});
</script>
</body> </body>

0
backend/.gitignore vendored Normal file → Executable file
View File

View File

@@ -1,28 +0,0 @@
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Copyright (c) Megvii, Inc. and its affiliates.
import os
from yolox.exp import Exp as MyExp
class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.data_dir = "/home/kitraining/To_Annotate/"
self.train_ann = "coco_project_27_train.json"
self.val_ann = "coco_project_27_valid.json"
self.test_ann = "coco_project_27_test.json"
self.num_classes = 80
self.pretrained_ckpt = r'/home/kitraining/Yolox/YOLOX-main/pretrained/YOLOX-s.pth'
self.depth = 1.0
self.width = 1.0
self.input_size = (640.0, 640.0)
self.mosaic_scale = (0.1, 2.0)
self.random_size = (10, 20)
self.test_size = (640.0, 640.0)
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]
self.enable_mixup = False

View File

@@ -1,48 +0,0 @@
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Copyright (c) Megvii, Inc. and its affiliates.
import os
from yolox.exp import Exp as MyExp
class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.data_dir = "/home/kitraining/To_Annotate/"
self.train_ann = "coco_project_2_train.json"
self.val_ann = "coco_project_2_valid.json"
self.test_ann = "coco_project_2_test.json"
self.num_classes = 2
self.pretrained_ckpt = r'/home/kitraining/Yolox/YOLOX-main/pretrained/YOLOX-s.pth'
self.activation = "silu"
self.depth = 0.33
self.scheduler = "yoloxwarmcos"
self.width = 0.5
self.input_size = (640.0, 640.0)
self.mosaic_scale = (0.1, 2.0)
self.test_size = (640.0, 640.0)
self.enable_mixup = True
self.max_epoch = 300
self.warmup_epochs = 5
self.warmup_lr = 0.0
self.no_aug_epochs = 15
self.min_lr_ratio = 0.05
self.ema = True
self.weight_decay = 0.0005
self.momentum = 0.9
self.print_interval = 10
self.eval_interval = 10
self.test_conf = 0.01
self.nms_thre = 0.65
self.mosaic_prob = 1.0
self.mixup_prob = 1.0
self.hsv_prob = 1.0
self.flip_prob = 0.5
self.degrees = 10.0
self.translate = 0.1
self.shear = 2.0
self.mixup_scale = (0.5, 1.5)
self.random_size = (10, 20)
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]

View File

@@ -1,48 +0,0 @@
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Copyright (c) Megvii, Inc. and its affiliates.
import os
from yolox.exp import Exp as MyExp
class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.data_dir = "/home/kitraining/To_Annotate/"
self.train_ann = "coco_project_6_train.json"
self.val_ann = "coco_project_6_valid.json"
self.test_ann = "coco_project_6_test.json"
self.num_classes = 2
self.pretrained_ckpt = r'/home/kitraining/Yolox/YOLOX-main/pretrained/YOLOX-s.pth'
self.depth = 0.33
self.width = 0.5
self.input_size = (640.0, 640.0)
self.mosaic_scale = (0.1, 2.0)
self.test_size = (640.0, 640.0)
self.enable_mixup = True
self.max_epoch = 300
self.warmup_epochs = 5
self.warmup_lr = 0.0
self.scheduler = "yoloxwarmcos"
self.no_aug_epochs = 15
self.min_lr_ratio = 0.05
self.ema = True
self.weight_decay = 0.0005
self.momentum = 0.9
self.print_interval = 10
self.eval_interval = 10
self.test_conf = 0.01
self.nms_thre = 0.65
self.mosaic_prob = 1.0
self.mixup_prob = 1.0
self.hsv_prob = 1.0
self.flip_prob = 0.5
self.degrees = 10.0
self.translate = 0.1
self.shear = 2.0
self.mixup_scale = (0.5, 1.5)
self.activation = "silu"
self.random_size = (10, 20)
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]

View File

@@ -1,113 +0,0 @@
# Quick Start Guide - Python Backend
## Step-by-Step Setup
### 1. Install Python
Make sure you have Python 3.8 or higher installed:
```bash
python --version
```
### 2. Create Virtual Environment
```bash
cd backend
python -m venv venv
```
### 3. Activate Virtual Environment
**Windows:**
```powershell
.\venv\Scripts\Activate.ps1
```
**Linux/Mac:**
```bash
source venv/bin/activate
```
### 4. Install Dependencies
```bash
pip install -r requirements.txt
```
### 5. Verify Database Connection
Make sure MySQL is running and the database `myapp` exists:
```sql
CREATE DATABASE IF NOT EXISTS myapp;
```
### 6. Run the Server
```bash
python start.py
```
Or:
```bash
python app.py
```
The server should now be running at `http://0.0.0.0:3000`
## Testing the API
Test if the server is working:
```bash
curl http://localhost:3000/api/training-projects
```
## Common Issues
### ModuleNotFoundError
If you get import errors, make sure you've activated the virtual environment and installed all dependencies.
### Database Connection Error
Check that:
- MySQL is running
- Database credentials in `app.py` are correct
- Database `myapp` exists
### Port Already in Use
If port 3000 is already in use, modify the port in `app.py`:
```python
app.run(host='0.0.0.0', port=3001, debug=True)
```
## What Changed from Node.js
1. **Server Framework**: Express.js → Flask
2. **ORM**: Sequelize → SQLAlchemy
3. **HTTP Client**: node-fetch → requests
4. **Package Manager**: npm → pip
5. **Dependencies**: package.json → requirements.txt
6. **Startup**: `node server.js``python app.py`
## Next Steps
1. Test all API endpoints
2. Update frontend to point to the new Python backend (if needed)
3. Migrate any remaining Node.js-specific logic
4. Test file uploads and downloads
5. Test YOLOX training functionality
## File Structure Comparison
**Before (Node.js):**
```
backend/
├── server.js
├── package.json
├── routes/api.js
├── models/*.js
└── services/*.js
```
**After (Python):**
```
backend/
├── app.py
├── requirements.txt
├── routes/api.py
├── models/*.py
└── services/*.py
```

0
backend/README.md Normal file → Executable file
View File

0
backend/app.py Normal file → Executable file
View File

View File

@@ -1,25 +0,0 @@
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Copyright (c) Megvii, Inc. and its affiliates.
import os
from yolox.exp import Exp as MyExp
class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.data_dir = "/home/kitraining/To_Annotate/"
self.train_ann = "coco_project_5_train.json"
self.val_ann = "coco_project_5_valid.json"
self.test_ann = "coco_project_5_test.json"
self.num_classes = 4
self.depth = 1.0
self.width = 1.0
self.input_size = (640, 640)
self.mosaic_scale = (0.1, 2)
self.random_size = (10, 20)
self.test_size = (640, 640)
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]
self.enable_mixup = False

View File

@@ -1,84 +0,0 @@
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Copyright (c) Megvii, Inc. and its affiliates.
import os
from yolox.exp import Exp as MyExp
class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.data_dir = "/home/kitraining/To_Annotate" # Where images are located
self.annotations_dir = "./backend/1/custom_exp_1" # Where annotation JSONs are located
self.train_ann = "coco_project_1_train.json"
self.val_ann = "coco_project_1_valid.json"
self.test_ann = "coco_project_1_test.json"
self.num_classes = 2
# Disable train2017 subdirectory - our images are directly in data_dir
self.name = ""
# Set data workers for training
self.data_num_workers = 8
self.depth = 1.0
self.width = 1.0
self.input_size = (640.0, 640.0)
self.mosaic_scale = (0.1, 2.0)
self.test_size = (640.0, 640.0)
self.enable_mixup = True
self.max_epoch = 300
self.warmup_epochs = 5
self.warmup_lr = 0.0
self.scheduler = "yoloxwarmcos"
self.no_aug_epochs = 15
self.min_lr_ratio = 0.05
self.ema = True
self.weight_decay = 0.0005
self.momentum = 0.9
self.print_interval = 10
self.eval_interval = 10
self.test_conf = 0.01
self.nms_thre = 0.65
self.mosaic_prob = 1.0
self.mixup_prob = 1.0
self.hsv_prob = 1.0
self.flip_prob = 0.5
self.degrees = 10.0
self.translate = 0.1
self.shear = 2.0
self.mixup_scale = (0.5, 1.5)
self.activation = "silu"
self.random_size = (10, 20)
def get_dataset(self, cache=False, cache_type="ram"):
"""Override to use name parameter for images directory"""
from yolox.data import COCODataset
# COCODataset constructs image paths as: os.path.join(data_dir, name, file_name)
# YOLOX adds "annotations/" to data_dir automatically, so we pass annotations_dir directly
# Use empty string for name since we have absolute paths in JSON
return COCODataset(
data_dir=self.annotations_dir,
json_file=self.train_ann,
name="",
img_size=self.input_size,
preproc=self.preproc if hasattr(self, 'preproc') else None,
cache=cache,
cache_type=cache_type,
)
def get_eval_dataset(self, **kwargs):
"""Override eval dataset using name parameter"""
from yolox.data import COCODataset
testdev = kwargs.get("testdev", False)
legacy = kwargs.get("legacy", False)
return COCODataset(
data_dir=self.annotations_dir,
json_file=self.val_ann if not testdev else self.test_ann,
name="",
img_size=self.test_size,
preproc=None, # No preprocessing for evaluation
)
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]

View File

@@ -1,84 +0,0 @@
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Copyright (c) Megvii, Inc. and its affiliates.
import os
from yolox.exp import Exp as MyExp
class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.data_dir = "/home/kitraining/To_Annotate" # Where images are located
self.annotations_dir = "./backend/1/custom_exp_1" # Where annotation JSONs are located
self.train_ann = "coco_project_1_train.json"
self.val_ann = "coco_project_1_valid.json"
self.test_ann = "coco_project_1_test.json"
self.num_classes = 2
# Disable train2017 subdirectory - our images are directly in data_dir
self.name = ""
# Set data workers for training
self.data_num_workers = 8
self.depth = 1.0
self.width = 1.0
self.input_size = (640.0, 640.0)
self.mosaic_scale = (0.1, 2.0)
self.test_size = (640.0, 640.0)
self.enable_mixup = True
self.max_epoch = 300
self.warmup_epochs = 5
self.warmup_lr = 0.0
self.scheduler = "yoloxwarmcos"
self.no_aug_epochs = 15
self.min_lr_ratio = 0.05
self.ema = True
self.weight_decay = 0.0005
self.momentum = 0.9
self.print_interval = 10
self.eval_interval = 10
self.test_conf = 0.01
self.nms_thre = 0.65
self.mosaic_prob = 1.0
self.mixup_prob = 1.0
self.hsv_prob = 1.0
self.flip_prob = 0.5
self.degrees = 10.0
self.translate = 0.1
self.shear = 2.0
self.mixup_scale = (0.5, 1.5)
self.activation = "silu"
self.random_size = (10, 20)
def get_dataset(self, cache=False, cache_type="ram"):
"""Override to use name parameter for images directory"""
from yolox.data import COCODataset
# COCODataset constructs image paths as: os.path.join(data_dir, name, file_name)
# YOLOX adds "annotations/" to data_dir automatically, so we pass annotations_dir directly
# Use empty string for name since we have absolute paths in JSON
return COCODataset(
data_dir=self.annotations_dir,
json_file=self.train_ann,
name="",
img_size=self.input_size,
preproc=self.preproc if hasattr(self, 'preproc') else None,
cache=cache,
cache_type=cache_type,
)
def get_eval_dataset(self, **kwargs):
"""Override eval dataset using name parameter"""
from yolox.data import COCODataset
testdev = kwargs.get("testdev", False)
legacy = kwargs.get("legacy", False)
return COCODataset(
data_dir=self.annotations_dir,
json_file=self.val_ann if not testdev else self.test_ann,
name="",
img_size=self.test_size,
preproc=None, # No preprocessing for evaluation
)
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]

0
backend/check_db.py Normal file → Executable file
View File

0
backend/data/README.md Normal file → Executable file
View File

0
backend/data/__init__.py Normal file → Executable file
View File

0
backend/data/test_base_configs.py Normal file → Executable file
View File

0
backend/data/yolox_l.py Normal file → Executable file
View File

0
backend/data/yolox_m.py Normal file → Executable file
View File

0
backend/data/yolox_s.py Normal file → Executable file
View File

0
backend/data/yolox_x.py Normal file → Executable file
View File

0
backend/database/__init__.py Normal file → Executable file
View File

View File

@@ -1,10 +0,0 @@
// database.js
const { Sequelize } = require('sequelize');
const sequelize = new Sequelize('myapp', 'root', 'root', {
host: 'localhost',
dialect: 'mysql',
logging: false,
});
module.exports = sequelize;

0
backend/database/database.py Normal file → Executable file
View File

View File

@@ -1,12 +0,0 @@
-- Migration: Add width and height columns to image table
-- Date: 2025-11-27
USE myapp;
-- Add width and height columns to image table
ALTER TABLE `image`
ADD COLUMN `width` FLOAT NULL AFTER `image_path`,
ADD COLUMN `height` FLOAT NULL AFTER `width`;
-- Verify the changes
DESCRIBE `image`;

View File

@@ -1,12 +0,0 @@
-- Migration to change width and height from FLOAT to INT in image table
-- Run this after updating the Images model
-- First, backup the table (optional but recommended)
-- CREATE TABLE image_backup AS SELECT * FROM image;
-- Alter the columns to INT type
ALTER TABLE image MODIFY COLUMN width INT;
ALTER TABLE image MODIFY COLUMN height INT;
-- Verify the changes
DESCRIBE image;

File diff suppressed because one or more lines are too long

View File

@@ -1,40 +0,0 @@
const { DataTypes } = require('sequelize');
const sequelize = require('../database/database.js');
const Annotation = sequelize.define('Annotation', {
annotation_id: {
type: DataTypes.INTEGER,
primaryKey: true,
autoIncrement: true,
},
image_id: {
type: DataTypes.INTEGER,
allowNull: false,
},
x: {
type: DataTypes.FLOAT,
allowNull: false,
},
y: {
type: DataTypes.FLOAT,
allowNull: false,
},
height: {
type: DataTypes.FLOAT,
allowNull: false,
},
width: {
type: DataTypes.FLOAT,
allowNull: false,
},
Label: {
type: DataTypes.STRING,
allowNull: false,
},
}, {
tableName: 'annotation',
timestamps: false,
});
module.exports = Annotation;

0
backend/models/Annotation.py Normal file → Executable file
View File

0
backend/models/AnnotationProjectMapping.py Normal file → Executable file
View File

0
backend/models/ClassMapping.py Normal file → Executable file
View File

View File

@@ -1,35 +0,0 @@
const { DataTypes } = require('sequelize');
const sequelize = require('../database/database.js');
const Image = sequelize.define('Image', {
image_id: {
type: DataTypes.INTEGER,
primaryKey: true,
autoIncrement: true,
},
image_path: {
type: DataTypes.STRING,
allowNull: false,
},
project_id: {
type: DataTypes.INTEGER,
allowNull: false,
},
width: {
type: DataTypes.FLOAT,
allowNull: true,
},
height: {
type: DataTypes.FLOAT,
allowNull: true,
},
}, {
tableName: 'image',
timestamps: false,
});
module.exports = Image;

0
backend/models/Images.py Normal file → Executable file
View File

View File

@@ -1,24 +0,0 @@
const { DataTypes } = require('sequelize');
const sequelize = require('../database/database.js');
const Label_studio_project = sequelize.define('LabelStudioProject', {
project_id: {
type: DataTypes.INTEGER,
primaryKey: true,
unique: true,
allowNull: false,
},
title:{
type: DataTypes.STRING,
allowNull: false,
}
}, {
tableName: 'label_studio_project',
timestamps: false,
});
module.exports = Label_studio_project;

0
backend/models/LabelStudioProject.py Normal file → Executable file
View File

0
backend/models/ProjectClass.py Normal file → Executable file
View File

0
backend/models/Settings.py Normal file → Executable file
View File

View File

@@ -1,38 +0,0 @@
const { DataTypes } = require('sequelize');
const sequelize = require('../database/database.js');
const Training_Project = sequelize.define('LabelStudioProject', {
project_id: {
type: DataTypes.INTEGER,
primaryKey: true,
unique: true,
allowNull: false,
autoIncrement: true,
},
title:{
type: DataTypes.STRING,
allowNull: false,
},
description: {
type: DataTypes.STRING,
},
classes: {
type: DataTypes.JSON,
allowNull: false,
},
project_image: {
type: DataTypes.BLOB,
},
project_image_type: {
type: DataTypes.STRING,
allowNull: true,
}
}, {
tableName: 'training_project',
timestamps: false,
});
module.exports = Training_Project;

0
backend/models/TrainingProject.py Normal file → Executable file
View File

View File

@@ -1,33 +0,0 @@
const { DataTypes } = require('sequelize');
const sequelize = require('../database/database.js');
const TrainingProjectDetails = sequelize.define('TrainingProjectDetails', {
id: {
type: DataTypes.INTEGER,
primaryKey: true,
autoIncrement: true,
unique: true,
},
project_id: {
type: DataTypes.INTEGER,
allowNull: false,
unique: true,
},
annotation_projects: {
type: DataTypes.JSON,
allowNull: false,
},
class_map: {
type: DataTypes.JSON,
allowNull: true,
},
description: {
type: DataTypes.JSON,
allowNull: true,
}
}, {
tableName: 'training_project_details',
timestamps: false,
});
module.exports = TrainingProjectDetails;

0
backend/models/TrainingProjectDetails.py Normal file → Executable file
View File

0
backend/models/TrainingSize.py Normal file → Executable file
View File

0
backend/models/__init__.py Normal file → Executable file
View File

View File

@@ -1,30 +0,0 @@
const LabelStudioProject = require('./LabelStudioProject.js');
const Annotation = require('./Annotation.js');
const Image = require('./Images.js');
const sequelize = require('../database/database.js');
const TrainingProjectDetails = require('./TrainingProjectDetails.js');
const TrainingProject = require('./TrainingProject.js');
const Training = require('./training.js');
const Project = LabelStudioProject;
const Img = Image;
const Ann = Annotation;
// Associations
Project.hasMany(Img, { foreignKey: 'project_id' });
Img.belongsTo(Project, { foreignKey: 'project_id' });
Img.hasMany(Ann, { foreignKey: 'image_id' });
Ann.belongsTo(Img, { foreignKey: 'image_id' });
// TrainingProjectDetails <-> TrainingProject
TrainingProjectDetails.belongsTo(TrainingProject, { foreignKey: 'project_id' });
TrainingProject.hasOne(TrainingProjectDetails, { foreignKey: 'project_id' });
// Training <-> TrainingProjectDetails
Training.belongsTo(TrainingProjectDetails, { foreignKey: 'project_details_id' });
TrainingProjectDetails.hasMany(Training, { foreignKey: 'project_details_id' });
module.exports = { Project, Img, Ann, TrainingProjectDetails, TrainingProject, Training };

View File

@@ -1,140 +0,0 @@
const { DataTypes } = require('sequelize');
const sequelize = require('../database/database.js');
const Training = sequelize.define('training', {
id: {
type: DataTypes.INTEGER,
autoIncrement: true,
unique: true,
primaryKey: true
},
exp_name: {
type: DataTypes.STRING(255)
},
max_epoch: {
type: DataTypes.INTEGER
},
depth: {
type: DataTypes.FLOAT
},
width: {
type: DataTypes.FLOAT
},
activation: {
type: DataTypes.STRING(255)
},
warmup_epochs: {
type: DataTypes.INTEGER
},
warmup_lr: {
type: DataTypes.FLOAT
},
basic_lr_per_img: {
type: DataTypes.FLOAT
},
scheduler: {
type: DataTypes.STRING(255)
},
no_aug_epochs: {
type: DataTypes.INTEGER
},
min_lr_ratio: {
type: DataTypes.FLOAT
},
ema: {
type: DataTypes.BOOLEAN
},
weight_decay: {
type: DataTypes.FLOAT
},
momentum: {
type: DataTypes.FLOAT
},
input_size: {
type: DataTypes.JSON
},
print_interval: {
type: DataTypes.INTEGER
},
eval_interval: {
type: DataTypes.INTEGER
},
save_history_ckpt: {
type: DataTypes.BOOLEAN
},
test_size: {
type: DataTypes.JSON
},
test_conf: {
type: DataTypes.FLOAT
},
nms_thre: {
type: DataTypes.FLOAT
},
multiscale_range: {
type: DataTypes.INTEGER
},
enable_mixup: {
type: DataTypes.BOOLEAN
},
mosaic_prob: {
type: DataTypes.FLOAT
},
mixup_prob: {
type: DataTypes.FLOAT
},
hsv_prob: {
type: DataTypes.FLOAT
},
flip_prob: {
type: DataTypes.FLOAT
},
degrees: {
type: DataTypes.FLOAT
},
mosaic_scale: {
type: DataTypes.JSON
},
mixup_scale: {
type: DataTypes.JSON
},
translate: {
type: DataTypes.FLOAT
},
shear: {
type: DataTypes.FLOAT
},
training_name: {
type: DataTypes.STRING(255)
},
project_details_id: {
type: DataTypes.INTEGER,
allowNull: false
},
seed: {
type: DataTypes.INTEGER
},
train: {
type: DataTypes.INTEGER
},
valid: {
type: DataTypes.INTEGER
},
test: {
type: DataTypes.INTEGER
},
selected_model: {
type: DataTypes.STRING(255)
},
transfer_learning: {
type: DataTypes.STRING(255)
},
model_upload: {
type: DataTypes.BLOB
}
}, {
tableName: 'training',
timestamps: false
});
module.exports = Training;

0
backend/models/training.py Normal file → Executable file
View File

0
backend/node Normal file → Executable file
View File

0
backend/package-lock.json generated Normal file → Executable file
View File

0
backend/package.json Normal file → Executable file
View File

0
backend/requirements.txt Normal file → Executable file
View File

0
backend/routes/__init__.py Normal file → Executable file
View File

View File

@@ -1,496 +0,0 @@
const express = require('express');
const multer = require('multer');
const upload = multer();
const TrainingProject = require('../models/TrainingProject.js');
const LabelStudioProject = require('../models/LabelStudioProject.js')
const { seedLabelStudio, updateStatus } = require('../services/seed-label-studio.js');
const fs = require('fs');
const path = require('path');
const {generateTrainingJson} = require('../services/generate-json-yolox.js')
const router = express.Router();
// Ensure JSON bodies are parsed for all routes
router.use(express.json());
router.get('/seed', async (req, res) => {
const result = await seedLabelStudio();
res.json(result);
});
// Trigger generate-json-yolox.js
router.post('/generate-yolox-json', async (req, res) => {
const { project_id } = req.body;
if (!project_id) {
return res.status(400).json({ message: 'Missing project_id in request body' });
}
try {
// Generate COCO JSONs
// Find all TrainingProjectDetails for this project
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const detailsRows = await TrainingProjectDetails.findAll({ where: { project_id } });
if (!detailsRows || detailsRows.length === 0) {
return res.status(404).json({ message: 'No TrainingProjectDetails found for project ' + project_id });
}
// For each details row, generate coco.jsons and exp.py in projectfolder/project_details_id
const Training = require('../models/training.js');
const { saveYoloxExp } = require('../services/generate-yolox-exp.js');
const TrainingProject = require('../models/TrainingProject.js');
const trainingProject = await TrainingProject.findByPk(project_id);
const projectName = trainingProject.name ? trainingProject.name.replace(/\s+/g, '_') : `project_${project_id}`;
for (const details of detailsRows) {
const detailsId = details.id;
await generateTrainingJson(detailsId);
const trainings = await Training.findAll({ where: { project_details_id: detailsId } });
if (trainings.length === 0) continue;
// For each training, save exp.py in projectfolder/project_details_id
const outDir = path.join(__dirname, '..', projectName, String(detailsId));
if (!fs.existsSync(outDir)) fs.mkdirSync(outDir, { recursive: true });
for (const training of trainings) {
const expFilePath = path.join(outDir, 'exp.py');
await saveYoloxExp(training.id, expFilePath);
}
}
// Find all trainings for this project
// ...existing code...
res.json({ message: 'YOLOX JSON and exp.py generated for project ' + project_id });
} catch (err) {
console.error('Error generating YOLOX JSON:', err);
res.status(500).json({ message: 'Failed to generate YOLOX JSON', error: err.message });
}
});
// Start YOLOX training
const { spawn } = require('child_process');
router.post('/start-yolox-training', async (req, res) => {
try {
const { project_id, training_id } = req.body;
// Get project name
const trainingProject = await TrainingProject.findByPk(project_id);
const projectName = trainingProject.name ? trainingProject.name.replace(/\s+/g, '_') : `project_${project_id}`;
// Look up training row by id or project_details_id
const Training = require('../models/training.js');
let trainingRow = await Training.findByPk(training_id);
if (!trainingRow) {
trainingRow = await Training.findOne({ where: { project_details_id: training_id } });
}
if (!trainingRow) {
return res.status(404).json({ error: `Training row not found for id or project_details_id ${training_id}` });
}
const project_details_id = trainingRow.project_details_id;
// Use the generated exp.py from the correct project folder
const outDir = path.join(__dirname, '..', projectName, String(project_details_id));
const yoloxMainDir = '/home/kitraining/Yolox/YOLOX-main';
const expSrc = path.join(outDir, 'exp.py');
if (!fs.existsSync(expSrc)) {
return res.status(500).json({ error: `exp.py not found at ${expSrc}` });
}
// Source venv and run YOLOX training in YOLOX-main folder
const yoloxVenv = '/home/kitraining/Yolox/yolox_venv/bin/activate';
// Determine model argument based on selected_model and transfer_learning
let modelArg = '';
let cmd = '';
if (
trainingRow.transfer_learning &&
typeof trainingRow.transfer_learning === 'string' &&
trainingRow.transfer_learning.toLowerCase() === 'coco'
) {
// If transfer_learning is 'coco', add -o and modelArg
modelArg = ` -c /home/kitraining/Yolox/YOLOX-main/pretrained/${trainingRow.selected_model}`;
cmd = `bash -c 'source ${yoloxVenv} && python tools/train.py -f ${expSrc} -d 1 -b 8 --fp16 -o ${modelArg}.pth --cache'`;
} else if (
trainingRow.selected_model &&
trainingRow.selected_model.toLowerCase() === 'coco' &&
(!trainingRow.transfer_learning || trainingRow.transfer_learning === false)
) {
// If selected_model is 'coco' and not transfer_learning, add modelArg only
modelArg = ` -c /pretrained/${trainingRow.selected_model}`;
cmd = `bash -c 'source ${yoloxVenv} && python tools/train.py -f ${expSrc} -d 1 -b 8 --fp16 -o ${modelArg}.pth --cache'`;
} else {
// Default: no modelArg
cmd = `bash -c 'source ${yoloxVenv} && python tools/train.py -f ${expSrc} -d 1 -b 8 --fp16' --cache`;
}
console.log(cmd)
const child = spawn(cmd, { shell: true, cwd: yoloxMainDir });
child.stdout.pipe(process.stdout);
child.stderr.pipe(process.stderr);
res.json({ message: 'Training started' });
} catch (err) {
res.status(500).json({ error: 'Failed to start training', details: err.message });
}
});
// Get YOLOX training log
router.get('/training-log', async (req, res) => {
try {
const { project_id, training_id } = req.query;
const trainingProject = await TrainingProject.findByPk(project_id);
const projectName = trainingProject.name ? trainingProject.name.replace(/\s+/g, '_') : `project_${project_id}`;
const outDir = path.join(__dirname, '..', projectName, String(training_id));
const logPath = path.join(outDir, 'training.log');
if (!fs.existsSync(logPath)) {
return res.status(404).json({ error: 'Log not found' });
}
const logData = fs.readFileSync(logPath, 'utf8');
res.json({ log: logData });
} catch (err) {
res.status(500).json({ error: 'Failed to fetch log', details: err.message });
}
});
router.post('/training-projects', upload.single('project_image'), async (req, res) => {
try {
const { title, description } = req.body;
const classes = JSON.parse(req.body.classes);
const project_image = req.file ? req.file.buffer : null;
const project_image_type = req.file ? req.file.mimetype : null;
await TrainingProject.create({
title,
description,
classes,
project_image,
project_image_type
});
res.json({ message: 'Project created!' });
} catch (error) {
console.error('Error creating project:', error);
res.status(500).json({ message: 'Failed to create project', error: error.message });
}
});
router.get('/training-projects', async (req, res) => {
try {
const projects = await TrainingProject.findAll();
// Convert BLOB to base64 data URL for each project
const serialized = projects.map(project => {
const plain = project.get({ plain: true });
if (plain.project_image) {
const base64 = Buffer.from(plain.project_image).toString('base64');
const mimeType = plain.project_image_type || 'image/png';
plain.project_image = `data:${mimeType};base64,${base64}`;
}
return plain;
});
res.json(serialized);
} catch (error) {
res.status(500).json({ message: 'Failed to fetch projects', error: error.message });
}
});
router.get('/update-status', async (req, res) => {
res.json(updateStatus)
})
router.get('/label-studio-projects', async (req, res) => {
try {
const LabelStudioProject = require('../models/LabelStudioProject.js');
const Image = require('../models/Images.js');
const Annotation = require('../models/Annotation.js');
const labelStudioProjects = await LabelStudioProject.findAll();
const projectsWithCounts = await Promise.all(labelStudioProjects.map(async project => {
const plain = project.get({ plain: true });
// Get all images for this project
const images = await Image.findAll({ where: { project_id: plain.project_id } });
let annotationCounts = {};
if (images.length > 0) {
const imageIds = images.map(img => img.image_id);
// Get all annotations for these images
const annotations = await Annotation.findAll({ where: { image_id: imageIds } });
// Count by label
for (const ann of annotations) {
const label = ann.Label;
annotationCounts[label] = (annotationCounts[label] || 0) + 1;
}
}
plain.annotationCounts = annotationCounts;
return plain;
}));
res.json(projectsWithCounts);
} catch (error) {
res.status(500).json({ message: 'Failed to fetch projects', error: error.message });
}
})
// POST endpoint to create TrainingProjectDetails with all fields
router.post('/training-project-details', async (req, res) => {
try {
const {
project_id,
annotation_projects,
class_map,
description
} = req.body;
if (!project_id || !annotation_projects) {
return res.status(400).json({ message: 'Missing required fields' });
}
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const created = await TrainingProjectDetails.create({
project_id,
annotation_projects,
class_map: class_map || null,
description: description || null
});
res.json({ message: 'TrainingProjectDetails created', details: created });
} catch (error) {
res.status(500).json({ message: 'Failed to create TrainingProjectDetails', error: error.message });
}
});
// GET endpoint to fetch all TrainingProjectDetails
router.get('/training-project-details', async (req, res) => {
try {
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const details = await TrainingProjectDetails.findAll();
res.json(details);
} catch (error) {
res.status(500).json({ message: 'Failed to fetch TrainingProjectDetails', error: error.message });
}
});
// PUT endpoint to update class_map and description in TrainingProjectDetails
router.put('/training-project-details', async (req, res) => {
try {
const { project_id, class_map, description } = req.body;
if (!project_id || !class_map || !description) {
return res.status(400).json({ message: 'Missing required fields' });
}
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const details = await TrainingProjectDetails.findOne({ where: { project_id } });
if (!details) {
return res.status(404).json({ message: 'TrainingProjectDetails not found' });
}
details.class_map = class_map;
details.description = description;
await details.save();
res.json({ message: 'Class map and description updated', details });
} catch (error) {
res.status(500).json({ message: 'Failed to update class map or description', error: error.message });
}
});
// POST endpoint to receive YOLOX settings and save to DB (handles multipart/form-data)
router.post('/yolox-settings', upload.any(), async (req, res) => {
try {
const settings = req.body;
// Debug: Log all received fields and types
console.log('--- YOLOX settings received ---');
console.log('settings:', settings);
if (req.files && req.files.length > 0) {
console.log('Files received:', req.files.map(f => ({ fieldname: f.fieldname, originalname: f.originalname, size: f.size })));
}
// Declare requiredFields once
const requiredFields = ['project_details_id', 'exp_name', 'max_epoch', 'depth', 'width', 'activation', 'train', 'valid', 'test', 'selected_model', 'transfer_learning'];
// Log types of required fields
requiredFields.forEach(field => {
console.log(`Field '${field}': value='${settings[field]}', type='${typeof settings[field]}'`);
});
// Map select_model to selected_model if present
if (settings && settings.select_model && !settings.selected_model) {
settings.selected_model = settings.select_model;
delete settings.select_model;
}
// Lookup project_details_id from project_id
if (!settings.project_id || isNaN(Number(settings.project_id))) {
throw new Error('Missing or invalid project_id in request. Cannot assign training to a project.');
}
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
let details = await TrainingProjectDetails.findOne({ where: { project_id: settings.project_id } });
if (!details) {
details = await TrainingProjectDetails.create({
project_id: settings.project_id,
annotation_projects: [],
class_map: null,
description: null
});
}
settings.project_details_id = details.id;
// Map 'act' from frontend to 'activation' for DB
if (settings.act !== undefined) {
settings.activation = settings.act;
delete settings.act;
}
// Type conversion for DB compatibility
[
'max_epoch', 'depth', 'width', 'warmup_epochs', 'warmup_lr', 'no_aug_epochs', 'min_lr_ratio', 'weight_decay', 'momentum', 'print_interval', 'eval_interval', 'test_conf', 'nmsthre', 'multiscale_range', 'degrees', 'translate', 'shear', 'train', 'valid', 'test'
].forEach(f => {
if (settings[f] !== undefined) settings[f] = Number(settings[f]);
});
// Improved boolean conversion
['ema', 'enable_mixup', 'save_history_ckpt'].forEach(f => {
if (settings[f] !== undefined) {
if (typeof settings[f] === 'string') {
settings[f] = settings[f].toLowerCase() === 'true';
} else {
settings[f] = Boolean(settings[f]);
}
}
});
// Improved array conversion
['mosaic_scale', 'mixup_scale', 'scale'].forEach(f => {
if (settings[f] && typeof settings[f] === 'string') {
settings[f] = settings[f]
.split(',')
.map(s => Number(s.trim()))
.filter(n => !isNaN(n));
}
});
// Trim all string fields
Object.keys(settings).forEach(f => {
if (typeof settings[f] === 'string') settings[f] = settings[f].trim();
});
// Set default for transfer_learning if missing
if (settings.transfer_learning === undefined) settings.transfer_learning = false;
// Convert empty string seed to null
if ('seed' in settings && (settings.seed === '' || settings.seed === undefined)) {
settings.seed = null;
}
// Validate required fields for training table
for (const field of requiredFields) {
if (settings[field] === undefined || settings[field] === null || settings[field] === '') {
console.error('Missing required field:', field, 'Value:', settings[field]);
throw new Error('Missing required field: ' + field);
}
}
console.log('Received YOLOX settings:', settings);
// Handle uploaded model file (ckpt_upload)
if (req.files && req.files.length > 0) {
const ckptFile = req.files.find(f => f.fieldname === 'ckpt_upload');
if (ckptFile) {
const uploadDir = path.join(__dirname, '..', 'uploads');
if (!fs.existsSync(uploadDir)) fs.mkdirSync(uploadDir);
const filename = ckptFile.originalname || `uploaded_model_${settings.project_id}.pth`;
const filePath = path.join(uploadDir, filename);
fs.writeFileSync(filePath, ckptFile.buffer);
settings.model_upload = filePath;
}
}
// Save settings to DB only (no file)
const { pushYoloxExpToDb } = require('../services/push-yolox-exp.js');
const training = await pushYoloxExpToDb(settings);
res.json({ message: 'YOLOX settings saved to DB', training });
} catch (error) {
console.error('Error in /api/yolox-settings:', error.stack || error);
res.status(500).json({ message: 'Failed to save YOLOX settings', error: error.message });
}
});
// POST endpoint to receive binary model file and save to disk (not DB)
router.post('/yolox-settings/upload', async (req, res) => {
try {
const projectId = req.query.project_id;
if (!projectId) return res.status(400).json({ message: 'Missing project_id in query' });
// Save file to disk
const uploadDir = path.join(__dirname, '..', 'uploads');
if (!fs.existsSync(uploadDir)) fs.mkdirSync(uploadDir);
const filename = req.headers['x-upload-filename'] || `uploaded_model_${projectId}.pth`;
const filePath = path.join(uploadDir, filename);
const chunks = [];
req.on('data', chunk => chunks.push(chunk));
req.on('end', async () => {
const buffer = Buffer.concat(chunks);
fs.writeFile(filePath, buffer, async err => {
if (err) {
console.error('Error saving file:', err);
return res.status(500).json({ message: 'Failed to save model file', error: err.message });
}
// Update latest training row for this project with file path
try {
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const Training = require('../models/training.js');
// Find details row for this project
const details = await TrainingProjectDetails.findOne({ where: { project_id: projectId } });
if (!details) return res.status(404).json({ message: 'No TrainingProjectDetails found for project_id' });
// Find latest training for this details row
const training = await Training.findOne({ where: { project_details_id: details.id }, order: [['createdAt', 'DESC']] });
if (!training) return res.status(404).json({ message: 'No training found for project_id' });
// Save file path to model_upload field
training.model_upload = filePath;
await training.save();
res.json({ message: 'Model file uploaded and saved to disk', filename, trainingId: training.id });
} catch (dbErr) {
console.error('Error updating training with file path:', dbErr);
res.status(500).json({ message: 'File saved but failed to update training row', error: dbErr.message });
}
});
});
} catch (error) {
console.error('Error in /api/yolox-settings/upload:', error.stack || error);
res.status(500).json({ message: 'Failed to upload model file', error: error.message });
}
});
// GET endpoint to fetch all trainings (optionally filtered by project_id)
router.get('/trainings', async (req, res) => {
try {
const project_id = req.query.project_id;
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const Training = require('../models/training.js');
if (project_id) {
// Find all details rows for this project
const detailsRows = await TrainingProjectDetails.findAll({ where: { project_id } });
if (!detailsRows || detailsRows.length === 0) return res.json([]);
// Get all trainings linked to any details row for this project
const detailsIds = detailsRows.map(d => d.id);
const trainings = await Training.findAll({ where: { project_details_id: detailsIds } });
return res.json(trainings);
} else {
// Return all trainings if no project_id is specified
const trainings = await Training.findAll();
return res.json(trainings);
}
} catch (error) {
res.status(500).json({ message: 'Failed to fetch trainings', error: error.message });
}
});
// DELETE endpoint to remove a training by id
router.delete('/trainings/:id', async (req, res) => {
try {
const Training = require('../models/training.js');
const id = req.params.id;
const deleted = await Training.destroy({ where: { id } });
if (deleted) {
res.json({ message: 'Training deleted' });
} else {
res.status(404).json({ message: 'Training not found' });
}
} catch (error) {
res.status(500).json({ message: 'Failed to delete training', error: error.message });
}
});
// DELETE endpoint to remove a training project and all related entries
router.delete('/training-projects/:id', async (req, res) => {
try {
const projectId = req.params.id;
const TrainingProject = require('../models/TrainingProject.js');
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const Training = require('../models/training.js');
// Find details row(s) for this project
const detailsRows = await TrainingProjectDetails.findAll({ where: { project_id: projectId } });
const detailsIds = detailsRows.map(d => d.id);
// Delete all trainings linked to these details
if (detailsIds.length > 0) {
await Training.destroy({ where: { project_details_id: detailsIds } });
await TrainingProjectDetails.destroy({ where: { project_id: projectId } });
}
// Delete the project itself
const deleted = await TrainingProject.destroy({ where: { project_id: projectId } });
if (deleted) {
res.json({ message: 'Training project and all related entries deleted' });
} else {
res.status(404).json({ message: 'Training project not found' });
}
} catch (error) {
res.status(500).json({ message: 'Failed to delete training project', error: error.message });
}
});
module.exports = router;

84
backend/routes/api.py Normal file → Executable file
View File

@@ -73,7 +73,7 @@ def generate_yolox_json():
@api_bp.route('/start-yolox-training', methods=['POST']) @api_bp.route('/start-yolox-training', methods=['POST'])
def start_yolox_training(): def start_yolox_training():
"""Generate JSONs, exp.py, and start YOLOX training""" """Generate JSONs, exp.py, and add training to queue"""
try: try:
data = request.get_json() data = request.get_json()
project_id = data.get('project_id') project_id = data.get('project_id')
@@ -90,9 +90,11 @@ def start_yolox_training():
details_id = training.project_details_id details_id = training.project_details_id
# Step 1: Generate COCO JSON files # Step 1: Generate COCO JSON files
# IMPORTANT: Pass training_id (not details_id) so JSON files are generated
# in the same location where exp.py expects them
from services.generate_json_yolox import generate_training_json from services.generate_json_yolox import generate_training_json
print(f'Generating COCO JSON for training {training_id}...') print(f'Generating COCO JSON for training {training_id}...')
generate_training_json(details_id) generate_training_json(training_id)
# Step 2: Generate exp.py # Step 2: Generate exp.py
from services.generate_yolox_exp import save_yolox_exp from services.generate_yolox_exp import save_yolox_exp
@@ -113,8 +115,8 @@ def start_yolox_training():
print(f'Generating exp.py at {exp_file_path}...') print(f'Generating exp.py at {exp_file_path}...')
save_yolox_exp(training_id, exp_file_path) save_yolox_exp(training_id, exp_file_path)
# Step 3: Start training # Step 3: Build training command
print(f'Starting YOLOX training for training {training_id}...') print(f'Preparing training command for training {training_id}...')
# Get YOLOX configuration from settings # Get YOLOX configuration from settings
yolox_main_dir = get_setting('yolox_path', '/home/kitraining/Yolox/YOLOX-main') yolox_main_dir = get_setting('yolox_path', '/home/kitraining/Yolox/YOLOX-main')
@@ -130,11 +132,17 @@ def start_yolox_training():
if (training.transfer_learning and if (training.transfer_learning and
isinstance(training.transfer_learning, str) and isinstance(training.transfer_learning, str) and
training.transfer_learning.lower() == 'coco'): training.transfer_learning.lower() == 'coco'):
model_arg = f'-c {yolox_main_dir}/pretrained/{training.selected_model}.pth' # Use yolox_path setting to construct pretrained model path
model_path = os.path.join(yolox_main_dir, 'pretrained', f'{training.selected_model}.pth')
model_path = model_path.replace('\\', '/') # Use forward slashes for command line
model_arg = f'-c {model_path}'
elif (training.selected_model and elif (training.selected_model and
training.selected_model.lower() == 'coco' and training.selected_model.lower() == 'coco' and
(not training.transfer_learning or training.transfer_learning == False)): (not training.transfer_learning or training.transfer_learning == False)):
model_arg = f'-c {yolox_main_dir}/pretrained/{training.selected_model}.pth' # Use yolox_path setting to construct pretrained model path
model_path = os.path.join(yolox_main_dir, 'pretrained', f'{training.selected_model}.pth')
model_path = model_path.replace('\\', '/') # Use forward slashes for command line
model_arg = f'-c {model_path}'
# Build base training arguments # Build base training arguments
train_args = f'-f {exp_file_path} -d 1 -b 8 --fp16 --cache' train_args = f'-f {exp_file_path} -d 1 -b 8 --fp16 --cache'
@@ -151,16 +159,18 @@ def start_yolox_training():
venv_activate = yolox_venv venv_activate = yolox_venv
cmd = f'cmd /c ""{venv_activate}" && python tools\\train.py {train_args}"' cmd = f'cmd /c ""{venv_activate}" && python tools\\train.py {train_args}"'
else: else:
# Linux: Use bash with source # Linux: Use bash with source to activate the venv
cmd = f'bash -c "source {yolox_venv} && python tools/train.py {train_args}"' venv_activate = os.path.join(yolox_venv, 'bin', 'activate')
cmd = f'bash -c "source {venv_activate} && python tools/train.py {train_args}"'
print(f'Training command: {cmd}') print(f'Training command: {cmd}')
# Start training in background # Step 4: Add to training queue
subprocess.Popen(cmd, shell=True, cwd=yolox_main_dir) from services.training_queue import training_queue
training_queue.add_to_queue(training_id, cmd, yolox_main_dir)
return jsonify({ return jsonify({
'message': f'JSONs and exp.py generated, training started for training {training_id}', 'message': f'Training {training_id} added to queue',
'exp_path': exp_file_path 'exp_path': exp_file_path
}) })
@@ -170,6 +180,18 @@ def start_yolox_training():
traceback.print_exc() traceback.print_exc()
return jsonify({'message': 'Failed to start training', 'error': str(err)}), 500 return jsonify({'message': 'Failed to start training', 'error': str(err)}), 500
@api_bp.route('/training-status', methods=['GET'])
def get_training_status():
"""Get current training queue status"""
try:
from services.training_queue import training_queue
status = training_queue.get_status()
return jsonify(status)
except Exception as err:
print(f'Error getting training status: {err}')
return jsonify({'current': None, 'queue': []}), 500
@api_bp.route('/training-log', methods=['GET']) @api_bp.route('/training-log', methods=['GET'])
def training_log(): def training_log():
"""Get YOLOX training log""" """Get YOLOX training log"""
@@ -477,17 +499,26 @@ def yolox_settings():
settings['activation'] = settings['act'] settings['activation'] = settings['act']
del settings['act'] del settings['act']
# Type conversions # Type conversions - Integer fields
numeric_fields = [ integer_fields = [
'max_epoch', 'depth', 'width', 'warmup_epochs', 'warmup_lr', 'max_epoch', 'warmup_epochs', 'no_aug_epochs', 'print_interval',
'no_aug_epochs', 'min_lr_ratio', 'weight_decay', 'momentum', 'eval_interval', 'multiscale_range', 'data_num_workers', 'num_classes'
'print_interval', 'eval_interval', 'test_conf', 'nmsthre',
'multiscale_range', 'degrees', 'translate', 'shear',
'train', 'valid', 'test'
] ]
for field in numeric_fields: for field in integer_fields:
if field in settings: if field in settings and settings[field] not in [None, '']:
settings[field] = int(float(settings[field])) # Convert via float to handle "10.0" strings
# Type conversions - Float fields
float_fields = [
'depth', 'width', 'warmup_lr', 'min_lr_ratio', 'weight_decay',
'momentum', 'test_conf', 'nmsthre', 'degrees', 'translate',
'shear', 'mosaic_prob', 'mixup_prob', 'hsv_prob', 'flip_prob',
'basic_lr_per_img', 'train', 'valid', 'test'
]
for field in float_fields:
if field in settings and settings[field] not in [None, '']:
settings[field] = float(settings[field]) settings[field] = float(settings[field])
# Boolean conversions # Boolean conversions
@@ -617,6 +648,19 @@ def get_trainings():
except Exception as error: except Exception as error:
return jsonify({'message': 'Failed to fetch trainings', 'error': str(error)}), 500 return jsonify({'message': 'Failed to fetch trainings', 'error': str(error)}), 500
@api_bp.route('/trainings/<int:id>', methods=['GET'])
def get_training(id):
"""Get a single training by id"""
try:
training = Training.query.get(id)
if training:
return jsonify(training.to_dict())
else:
return jsonify({'message': 'Training not found'}), 404
except Exception as error:
return jsonify({'message': 'Failed to fetch training', 'error': str(error)}), 500
@api_bp.route('/trainings/<int:id>', methods=['DELETE']) @api_bp.route('/trainings/<int:id>', methods=['DELETE'])
def delete_training(id): def delete_training(id):
"""Delete a training by id""" """Delete a training by id"""

0
backend/server.js Normal file → Executable file
View File

0
backend/services/__init__.py Normal file → Executable file
View File

View File

@@ -1,92 +0,0 @@
const API_URL = 'http://192.168.1.19:8080/api';
const API_TOKEN = 'c1cef980b7c73004f4ee880a42839313b863869f';
const fetch = require('node-fetch');
async function fetchLableStudioProject(projectid) {
// 1. Trigger export
const exportUrl = `${API_URL}/projects/${projectid}/export?exportType=JSON_MIN`;
const headers = { Authorization: `Token ${API_TOKEN}` };
let res = await fetch(exportUrl, { headers });
if (!res.ok) {
let errorText = await res.text().catch(() => '');
console.error(`Failed to trigger export: ${res.status} ${res.statusText} - ${errorText}`);
throw new Error(`Failed to trigger export: ${res.status} ${res.statusText}`);
}
let data = await res.json();
// If data is an array, it's ready
if (Array.isArray(data)) return data;
// If not, poll for the export file
let fileUrl = data.download_url || data.url || null;
let tries = 0;
while (!fileUrl && tries < 20) {
await new Promise(r => setTimeout(r, 2000));
res = await fetch(exportUrl, { headers });
if (!res.ok) {
let errorText = await res.text().catch(() => '');
console.error(`Failed to poll export: ${res.status} ${res.statusText} - ${errorText}`);
throw new Error(`Failed to poll export: ${res.status} ${res.statusText}`);
}
data = await res.json();
fileUrl = data.download_url || data.url || null;
tries++;
}
if (!fileUrl) throw new Error('Label Studio export did not become ready');
// 2. Download the export file
res = await fetch(fileUrl.startsWith('http') ? fileUrl : `${API_URL.replace('/api','')}${fileUrl}`, { headers });
if (!res.ok) {
let errorText = await res.text().catch(() => '');
console.error(`Failed to download export: ${res.status} ${res.statusText} - ${errorText}`);
throw new Error(`Failed to download export: ${res.status} ${res.statusText}`);
}
return await res.json();
}
async function fetchProjectIdsAndTitles() {
try {
const response = await fetch(`${API_URL}/projects/`, {
headers: {
'Authorization': `Token ${API_TOKEN}`,
'Content-Type': 'application/json'
}
});
if (!response.ok) {
let errorText = await response.text().catch(() => '');
console.error(`Failed to fetch projects: ${response.status} ${response.statusText} - ${errorText}`);
throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();
if (!data.results || !Array.isArray(data.results)) {
throw new Error('API response does not contain results array');
}
// Extract id and title from each project
const projects = data.results.map(project => ({
id: project.id,
title: project.title
}));
console.log(projects)
return projects;
} catch (error) {
console.error('Failed to fetch projects:', error);
return [];
}
}
module.exports = { fetchLableStudioProject, fetchProjectIdsAndTitles };
//getLableStudioProject(20)
//fetchProjectIdsAndTitles()

0
backend/services/fetch_labelstudio.py Normal file → Executable file
View File

View File

@@ -1,176 +0,0 @@
const TrainingProject = require('../models/TrainingProject.js');
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js')
const LabelStudioProject = require('../models/LabelStudioProject.js')
const Annotation = require('../models/Annotation.js')
const Images = require('../models/Images.js')
const fs = require('fs');
async function generateTrainingJson(trainingId){
// trainingId is now project_details_id
const trainingProjectDetails = await TrainingProjectDetails.findByPk(trainingId);
if (!trainingProjectDetails) throw new Error('No TrainingProjectDetails found for project_details_id ' + trainingId);
const detailsObj = trainingProjectDetails.get({ plain: true });
// Get parent project for name
const trainingProject = await TrainingProject.findByPk(detailsObj.project_id);
// Get split percentages (assume they are stored as train_percent, valid_percent, test_percent)
const trainPercent = detailsObj.train_percent || 85;
const validPercent = detailsObj.valid_percent || 10;
const testPercent = detailsObj.test_percent || 5;
let cocoImages = [];
let cocoAnnotations = [];
let cocoCategories = [];
let categoryMap = {};
let categoryId = 0;
let imageid = 0;
let annotationid = 0;
for (const cls of detailsObj.class_map) {
const asgMap = [];
const listAsg = cls[1];
for(const asg of listAsg){
asgMap.push ({ original: asg[0], mapped: asg[1] });
// Build category list and mapping
if (asg[1] && !(asg[1] in categoryMap)) {
categoryMap[asg[1]] = categoryId;
cocoCategories.push({ id: categoryId, name: asg[1], supercategory: '' });
categoryId++;
}
}
const images = await Images.findAll({ where: { project_id: cls[0] } });
for(const image of images){
imageid += 1;
let fileName = image.image_path;
if (fileName.includes('%20')) {
fileName = fileName.replace(/%20/g, ' ');
}
if (fileName && fileName.startsWith('/data/local-files/?d=')) {
fileName = fileName.replace('/data/local-files/?d=', '');
fileName = fileName.replace('/home/kitraining/home/kitraining/', '');
}
if (fileName && fileName.startsWith('home/kitraining/To_Annotate/')) {
fileName = fileName.replace('home/kitraining/To_Annotate/','');
}
// Get annotations for this image
const annotations = await Annotation.findAll({ where: { image_id: image.image_id } });
// Use image.width and image.height from DB (populated from original_width/original_height)
cocoImages.push({
id: imageid,
file_name: fileName,
width: image.width || 0,
height: image.height || 0
});
for (const annotation of annotations) {
// Translate class name using asgMap
let mappedClass = annotation.Label;
for (const mapEntry of asgMap) {
if (annotation.Label === mapEntry.original) {
mappedClass = mapEntry.mapped;
break;
}
}
// Only add annotation if mappedClass is valid
if (mappedClass && mappedClass in categoryMap) {
annotationid += 1;
let area = 0;
if (annotation.width && annotation.height) {
area = annotation.width * annotation.height;
}
cocoAnnotations.push({
id: annotationid,
image_id: imageid,
category_id: categoryMap[mappedClass],
bbox: [annotation.x, annotation.y, annotation.width, annotation.height],
area: area,
iscrowd: annotation.iscrowd || 0
});
}
}
}
}
// Shuffle images for random split using seed
function seededRandom(seed) {
let x = Math.sin(seed++) * 10000;
return x - Math.floor(x);
}
function shuffle(array, seed) {
for (let i = array.length - 1; i > 0; i--) {
const j = Math.floor(seededRandom(seed + i) * (i + 1));
[array[i], array[j]] = [array[j], array[i]];
}
}
// Use seed from detailsObj if present, else default to 42
const splitSeed = detailsObj.seed !== undefined && detailsObj.seed !== null ? Number(detailsObj.seed) : 42;
shuffle(cocoImages, splitSeed);
// Split images
const totalImages = cocoImages.length;
const trainCount = Math.floor(totalImages * trainPercent / 100);
const validCount = Math.floor(totalImages * validPercent / 100);
const testCount = totalImages - trainCount - validCount;
const trainImages = cocoImages.slice(0, trainCount);
const validImages = cocoImages.slice(trainCount, trainCount + validCount);
const testImages = cocoImages.slice(trainCount + validCount);
// Helper to get image ids for each split
const trainImageIds = new Set(trainImages.map(img => img.id));
const validImageIds = new Set(validImages.map(img => img.id));
const testImageIds = new Set(testImages.map(img => img.id));
// Split annotations
const trainAnnotations = cocoAnnotations.filter(ann => trainImageIds.has(ann.image_id));
const validAnnotations = cocoAnnotations.filter(ann => validImageIds.has(ann.image_id));
const testAnnotations = cocoAnnotations.filter(ann => testImageIds.has(ann.image_id));
// Build final COCO JSONs with info section
const buildCocoJson = (images, annotations, categories) => ({
images,
annotations,
categories
});
// Build COCO JSONs with info section
const trainJson = buildCocoJson(trainImages, trainAnnotations, cocoCategories);
const validJson = buildCocoJson(validImages, validAnnotations, cocoCategories);
const testJson = buildCocoJson(testImages, testAnnotations, cocoCategories);
// Create output directory: projectname/trainingid/annotations
const projectName = trainingProject && trainingProject.name ? trainingProject.name.replace(/\s+/g, '_') : `project_${detailsObj.project_id}`;
const outDir = `${projectName}/${trainingId}`;
const annotationsDir = `/home/kitraining/To_Annotate/annotations`;
if (!fs.existsSync(annotationsDir)) {
fs.mkdirSync(annotationsDir, { recursive: true });
}
// Write to files in the annotations directory
const trainPath = `${annotationsDir}/coco_project_${trainingId}_train.json`;
const validPath = `${annotationsDir}/coco_project_${trainingId}_valid.json`;
const testPath = `${annotationsDir}/coco_project_${trainingId}_test.json`;
fs.writeFileSync(trainPath, JSON.stringify(trainJson, null, 2));
fs.writeFileSync(validPath, JSON.stringify(validJson, null, 2));
fs.writeFileSync(testPath, JSON.stringify(testJson, null, 2));
console.log(`COCO JSON splits written to ${annotationsDir} for trainingId ${trainingId}`);
// Also generate inference exp.py in the same output directory as exp.py (project folder in workspace)
const { generateYoloxInferenceExp } = require('./generate-yolox-exp');
const path = require('path');
const projectFolder = path.join(__dirname, '..', projectName, String(trainingId));
if (!fs.existsSync(projectFolder)) {
fs.mkdirSync(projectFolder, { recursive: true });
}
const inferenceExpPath = path.join(projectFolder, 'exp_infer.py');
generateYoloxInferenceExp(trainingId).then(expContent => {
fs.writeFileSync(inferenceExpPath, expContent);
console.log(`Inference exp.py written to ${inferenceExpPath}`);
}).catch(err => {
console.error('Failed to generate inference exp.py:', err);
});
}
module.exports = {generateTrainingJson};

View File

@@ -1,135 +0,0 @@
const fs = require('fs');
const path = require('path');
const Training = require('../models/training.js');
const TrainingProject = require('../models/TrainingProject.js');
// Remove Python comments and legacy code
const exp_names = [
'YOLOX-s',
'YOLOX-m',
'YOLOX-l',
'YOLOX-x',
'YOLOX-Darknet53', //todo
'YOLOX-Nano',
'YOLOX-Tiny'
]
//TODO: Clean up generation of exp_names.py and remove second exp creation!!!
// Refactored: Accept trainingId, fetch info from DB
async function generateYoloxExp(trainingId) {
// Fetch training row from DB by project_details_id if not found by PK
let training = await Training.findByPk(trainingId);
if (!training) {
training = await Training.findOne({ where: { project_details_id: trainingId } });
}
if (!training) throw new Error('Training not found for trainingId or project_details_id: ' + trainingId);
// If transfer_learning is 'coco', just return the path to the default exp.py
if (training.transfer_learning === 'coco') {
const selectedModel = training.selected_model.toLowerCase().replace('-', '_');
const expSourcePath = `/home/kitraining/Yolox/YOLOX-main/exps/default/${selectedModel}.py`;
if (!fs.existsSync(expSourcePath)) {
throw new Error(`Default exp.py not found for model: ${selectedModel} at ${expSourcePath}`);
}
// Copy to project folder (e.g., /home/kitraining/coco_tool/backend/project_XX/YY/exp.py)
const projectDetailsId = training.project_details_id;
const projectFolder = path.resolve(__dirname, `../project_23/${projectDetailsId}`);
if (!fs.existsSync(projectFolder)) {
fs.mkdirSync(projectFolder, { recursive: true });
}
const expDestPath = path.join(projectFolder, 'exp.py');
fs.copyFileSync(expSourcePath, expDestPath);
return { type: 'default', expPath: expDestPath };
}
// If transfer_learning is 'sketch', generate a custom exp.py as before
if (training.transfer_learning === 'sketch') {
// ...existing custom exp.py generation logic here (copy from previous implementation)...
// For brevity, you can call generateYoloxInferenceExp or similar here, or inline the logic.
// Example:
const expContent = await generateYoloxInferenceExp(trainingId);
return { type: 'custom', expContent };
}
throw new Error('Unknown transfer_learning type: ' + training.transfer_learning);
}
async function saveYoloxExp(trainingId, outPath) {
const expResult = await generateYoloxExp(trainingId);
if (expResult.type === 'custom' && expResult.expContent) {
fs.writeFileSync(outPath, expResult.expContent);
return outPath;
} else if (expResult.type === 'default' && expResult.expPath) {
// Optionally copy the file if outPath is different
if (expResult.expPath !== outPath) {
fs.copyFileSync(expResult.expPath, outPath);
}
return outPath;
} else {
throw new Error('Unknown expResult type or missing content');
}
}
// Generate a second exp.py for inference, using the provided template and DB values
async function generateYoloxInferenceExp(trainingId, options = {}) {
let training = await Training.findByPk(trainingId);
if (!training) {
training = await Training.findOne({ where: { project_details_id: trainingId } });
}
if (!training) throw new Error('Training not found for trainingId or project_details_id: ' + trainingId);
// Always use the trainingId (project_details_id) for annotation file names
const projectDetailsId = training.project_details_id;
const dataDir = options.data_dir || '/home/kitraining/To_Annotate/';
const trainAnn = options.train_ann || `coco_project_${trainingId}_train.json`;
const valAnn = options.val_ann || `coco_project_${trainingId}_valid.json`;
const testAnn = options.test_ann || `coco_project_${trainingId}_test.json`;
// Get num_classes from TrainingProject.classes JSON
let numClasses = 80;
try {
const trainingProject = await TrainingProject.findByPk(projectDetailsId);
if (trainingProject && trainingProject.classes) {
let classesArr = trainingProject.classes;
if (typeof classesArr === 'string') {
classesArr = JSON.parse(classesArr);
}
if (Array.isArray(classesArr)) {
numClasses = classesArr.filter(c => c !== null && c !== undefined && c !== '').length;
} else if (typeof classesArr === 'object' && classesArr !== null) {
numClasses = Object.keys(classesArr).filter(k => classesArr[k] !== null && classesArr[k] !== undefined && classesArr[k] !== '').length;
}
}
} catch (e) {
console.warn('Could not determine num_classes from TrainingProject.classes:', e);
}
const depth = options.depth || training.depth || 1.00;
const width = options.width || training.width || 1.00;
const inputSize = options.input_size || training.input_size || [640, 640];
const mosaicScale = options.mosaic_scale || training.mosaic_scale || [0.1, 2];
const randomSize = options.random_size || training.random_size || [10, 20];
const testSize = options.test_size || training.test_size || [640, 640];
const expName = options.exp_name || 'inference_exp';
const enableMixup = options.enable_mixup !== undefined ? options.enable_mixup : false;
let expContent = '';
expContent += `#!/usr/bin/env python3\n# -*- coding:utf-8 -*-\n# Copyright (c) Megvii, Inc. and its affiliates.\n\nimport os\n\nfrom yolox.exp import Exp as MyExp\n\n\nclass Exp(MyExp):\n def __init__(self):\n super(Exp, self).__init__()\n self.data_dir = "${dataDir}"\n self.train_ann = "${trainAnn}"\n self.val_ann = "${valAnn}"\n self.test_ann = "coco_project_${trainingId}_test.json"\n self.num_classes = ${numClasses}\n`;
// Set pretrained_ckpt if transfer_learning is 'coco'
if (training.transfer_learning && typeof training.transfer_learning === 'string' && training.transfer_learning.toLowerCase() === 'coco') {
const yoloxBaseDir = '/home/kitraining/Yolox/YOLOX-main';
const selectedModel = training.selected_model ? training.selected_model.replace(/\.pth$/i, '') : '';
if (selectedModel) {
expContent += ` self.pretrained_ckpt = r'${yoloxBaseDir}/pretrained/${selectedModel}.pth'\n`;
}
}
expContent += ` self.depth = ${depth}\n self.width = ${width}\n self.input_size = (${Array.isArray(inputSize) ? inputSize.join(', ') : inputSize})\n self.mosaic_scale = (${Array.isArray(mosaicScale) ? mosaicScale.join(', ') : mosaicScale})\n self.random_size = (${Array.isArray(randomSize) ? randomSize.join(', ') : randomSize})\n self.test_size = (${Array.isArray(testSize) ? testSize.join(', ') : testSize})\n self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]\n self.enable_mixup = ${enableMixup ? 'True' : 'False'}\n`;
return expContent;
}
// Save inference exp.py to a custom path
async function saveYoloxInferenceExp(trainingId, outPath, options = {}) {
const expContent = await generateYoloxInferenceExp(trainingId, options);
fs.writeFileSync(outPath, expContent);
return outPath;
}
module.exports = { generateYoloxExp, saveYoloxExp, generateYoloxInferenceExp, saveYoloxInferenceExp };

85
backend/services/generate_json_yolox.py Normal file → Executable file
View File

@@ -7,12 +7,30 @@ from models.Images import Image
from models.Annotation import Annotation from models.Annotation import Annotation
def generate_training_json(training_id): def generate_training_json(training_id):
"""Generate COCO JSON for training, validation, and test sets""" """Generate COCO JSON for training, validation, and test sets
# training_id is now project_details_id
training_project_details = TrainingProjectDetails.query.get(training_id) Args:
training_id: Can be either a Training.id or TrainingProjectDetails.id
Function will automatically detect which one and find the correct details_id
"""
from models.training import Training
# First, try to get as a Training record
training_record = Training.query.get(training_id)
if training_record:
# It's a Training.id - use its project_details_id
details_id = training_record.project_details_id
print(f'[generate_training_json] Using training_id={training_id}, mapped to project_details_id={details_id}')
else:
# Try as TrainingProjectDetails.id directly
details_id = training_id
print(f'[generate_training_json] Using training_id={training_id} as project_details_id directly')
training_project_details = TrainingProjectDetails.query.get(details_id)
if not training_project_details: if not training_project_details:
raise Exception(f'No TrainingProjectDetails found for project_details_id {training_id}') raise Exception(f'No TrainingProjectDetails found for id {training_id} (details_id: {details_id})')
details_obj = training_project_details.to_dict() details_obj = training_project_details.to_dict()
@@ -110,22 +128,35 @@ def generate_training_json(training_id):
break break
# Construct ABSOLUTE path using data_dir # Construct ABSOLUTE path using data_dir
# Normalize data_dir - ensure it uses backslashes for Windows # Detect platform for proper path handling
normalized_data_dir = data_dir.rstrip('/\\').replace('/', '\\') import platform
is_windows = platform.system() == 'Windows'
# Normalize data_dir and file_name based on platform
if is_windows:
# Windows: use backslashes
normalized_data_dir = data_dir.rstrip('/\\').replace('/', '\\')
file_name = file_name.replace('/', '\\')
else:
# Linux/Mac: use forward slashes
normalized_data_dir = data_dir.rstrip('/\\').replace('\\', '/')
file_name = file_name.replace('\\', '/')
# Check if already absolute path # Check if already absolute path
if not (file_name.startswith('\\\\') or (len(file_name) > 1 and file_name[1] == ':')): is_absolute = False
# It's a relative path, combine with data_dir if is_windows:
# For UNC paths, we need to manually concatenate to preserve \\ is_absolute = file_name.startswith('\\\\') or (len(file_name) > 1 and file_name[1] == ':')
if normalized_data_dir.startswith('\\\\'):
# UNC path
file_name = normalized_data_dir + '\\' + file_name.replace('/', '\\')
else:
# Regular path
file_name = os.path.join(normalized_data_dir, file_name.replace('/', '\\'))
else: else:
# Already absolute, just normalize separators is_absolute = file_name.startswith('/')
file_name = file_name.replace('/', '\\')
if not is_absolute:
# It's a relative path, combine with data_dir
if is_windows and normalized_data_dir.startswith('\\\\'):
# Windows UNC path
file_name = normalized_data_dir + '\\' + file_name
else:
# Regular path (Windows or Linux)
file_name = os.path.join(normalized_data_dir, file_name)
# Get annotations for this image # Get annotations for this image
annotations = Annotation.query.filter_by(image_id=image.image_id).all() annotations = Annotation.query.filter_by(image_id=image.image_id).all()
@@ -218,13 +249,19 @@ def generate_training_json(training_id):
project_name = training_project.title.replace(' ', '_') if training_project and training_project.title else f'project_{details_obj["project_id"]}' project_name = training_project.title.replace(' ', '_') if training_project and training_project.title else f'project_{details_obj["project_id"]}'
# Get training record to use its name for folder # Get training record to use its name and ID for folder and file names
training_record = Training.query.filter_by(project_details_id=training_id).first() # Use the same training_id that was passed in (if it was a Training.id)
training_folder_name = f"{training_record.exp_name or training_record.training_name or 'training'}_{training_record.id}" if training_record else str(training_id) # or find the first training for this details_id
training_folder_name = training_folder_name.replace(' ', '_') if not training_record:
training_record = Training.query.filter_by(project_details_id=details_id).first()
# Use training_record.id for file names to match what generate_yolox_exp expects if training_record:
training_file_id = training_record.id if training_record else training_id training_folder_name = f"{training_record.exp_name or training_record.training_name or 'training'}_{training_record.id}"
training_folder_name = training_folder_name.replace(' ', '_')
training_file_id = training_record.id
else:
training_folder_name = str(details_id)
training_file_id = details_id
# Save annotations to the configured output folder # Save annotations to the configured output folder
annotations_dir = os.path.join(output_base_path, project_name, training_folder_name, 'annotations') annotations_dir = os.path.join(output_base_path, project_name, training_folder_name, 'annotations')
@@ -242,7 +279,7 @@ def generate_training_json(training_id):
with open(test_path, 'w') as f: with open(test_path, 'w') as f:
json.dump(test_json, f, indent=2) json.dump(test_json, f, indent=2)
print(f'COCO JSON splits written to {annotations_dir} for trainingId {training_id}') print(f'COCO JSON splits written to {annotations_dir} for training_id={training_file_id} (details_id={details_id})')
# Also generate inference exp.py # Also generate inference exp.py
from services.generate_yolox_exp import generate_yolox_inference_exp from services.generate_yolox_exp import generate_yolox_inference_exp

68
backend/services/generate_yolox_exp.py Normal file → Executable file
View File

@@ -220,6 +220,10 @@ def generate_yolox_inference_exp(training_id, options=None, use_base_config=Fals
annotations_parent_dir = os.path.join(output_base_path, project_name, training_folder_name) annotations_parent_dir = os.path.join(output_base_path, project_name, training_folder_name)
annotations_parent_escaped = annotations_parent_dir.replace('\\', '\\\\') annotations_parent_escaped = annotations_parent_dir.replace('\\', '\\\\')
# Set output directory for checkpoints - models subdirectory
models_dir = os.path.join(annotations_parent_dir, 'models')
models_dir_escaped = models_dir.replace('\\', '\\\\')
# Build exp content # Build exp content
exp_content = f'''#!/usr/bin/env python3 exp_content = f'''#!/usr/bin/env python3
# -*- coding:utf-8 -*- # -*- coding:utf-8 -*-
@@ -235,6 +239,7 @@ class Exp(MyExp):
super(Exp, self).__init__() super(Exp, self).__init__()
self.data_dir = "{data_dir_escaped}" # Where images are located self.data_dir = "{data_dir_escaped}" # Where images are located
self.annotations_dir = "{annotations_parent_escaped}" # Where annotation JSONs are located self.annotations_dir = "{annotations_parent_escaped}" # Where annotation JSONs are located
self.output_dir = "{models_dir_escaped}" # Where checkpoints will be saved
self.train_ann = "{train_ann}" self.train_ann = "{train_ann}"
self.val_ann = "{val_ann}" self.val_ann = "{val_ann}"
self.test_ann = "{test_ann}" self.test_ann = "{test_ann}"
@@ -252,21 +257,46 @@ class Exp(MyExp):
if selected_model: if selected_model:
exp_content += f" self.pretrained_ckpt = r'{yolox_base_dir}/pretrained/{selected_model}.pth'\n" exp_content += f" self.pretrained_ckpt = r'{yolox_base_dir}/pretrained/{selected_model}.pth'\n"
# Format arrays # Format arrays and values for Python code generation
def format_value(val): # Integer-only parameters (sizes, epochs, intervals)
integer_params = {
'input_size', 'test_size', 'random_size', 'max_epoch', 'warmup_epochs',
'no_aug_epochs', 'print_interval', 'eval_interval', 'multiscale_range',
'data_num_workers', 'num_classes'
}
def format_value(val, param_name=''):
if isinstance(val, (list, tuple)): if isinstance(val, (list, tuple)):
return '(' + ', '.join(map(str, val)) + ')' # Check if this parameter should have integer values
if param_name in integer_params:
# Convert all values to integers
formatted_items = [str(int(float(item))) if isinstance(item, (int, float)) else str(item) for item in val]
else:
# Keep as floats or original type
formatted_items = []
for item in val:
if isinstance(item, float):
formatted_items.append(str(item))
elif isinstance(item, int):
formatted_items.append(str(item))
else:
formatted_items.append(str(item))
return '(' + ', '.join(formatted_items) + ')'
elif isinstance(val, bool): elif isinstance(val, bool):
return str(val) return str(val)
elif isinstance(val, str): elif isinstance(val, str):
return f'"{val}"' return f'"{val}"'
elif isinstance(val, int):
return str(val)
elif isinstance(val, float):
return str(val)
else: else:
return str(val) return str(val)
# Add all config parameters to exp # Add all config parameters to exp
for key, value in config.items(): for key, value in config.items():
if key not in ['exp_name']: # exp_name is handled separately if key not in ['exp_name']: # exp_name is handled separately
exp_content += f" self.{key} = {format_value(value)}\n" exp_content += f" self.{key} = {format_value(value, key)}\n"
# Add get_dataset override using name parameter for image directory # Add get_dataset override using name parameter for image directory
exp_content += ''' exp_content += '''
@@ -289,7 +319,7 @@ class Exp(MyExp):
def get_eval_dataset(self, **kwargs): def get_eval_dataset(self, **kwargs):
"""Override eval dataset using name parameter""" """Override eval dataset using name parameter"""
from yolox.data import COCODataset from yolox.data import COCODataset, ValTransform
testdev = kwargs.get("testdev", False) testdev = kwargs.get("testdev", False)
legacy = kwargs.get("legacy", False) legacy = kwargs.get("legacy", False)
@@ -299,8 +329,34 @@ class Exp(MyExp):
json_file=self.val_ann if not testdev else self.test_ann, json_file=self.val_ann if not testdev else self.test_ann,
name="", name="",
img_size=self.test_size, img_size=self.test_size,
preproc=None, # No preprocessing for evaluation preproc=ValTransform(legacy=legacy), # Use proper validation transform
) )
def get_eval_loader(self, batch_size, is_distributed, **kwargs):
"""Standard YOLOX eval loader - matches official implementation"""
import torch
import torch.distributed as dist
from torch.utils.data import DataLoader
valdataset = self.get_eval_dataset(**kwargs)
if is_distributed:
batch_size = batch_size // dist.get_world_size()
sampler = torch.utils.data.distributed.DistributedSampler(
valdataset, shuffle=False
)
else:
sampler = torch.utils.data.SequentialSampler(valdataset)
dataloader_kwargs = {
"num_workers": self.data_num_workers,
"pin_memory": True,
"sampler": sampler,
}
dataloader_kwargs["batch_size"] = batch_size
val_loader = DataLoader(valdataset, **dataloader_kwargs)
return val_loader
''' '''
# Add exp_name at the end (uses dynamic path) # Add exp_name at the end (uses dynamic path)

View File

@@ -1,48 +0,0 @@
const Training = require('../models/training.js');
const fs = require('fs');
const path = require('path');
async function pushYoloxExpToDb(settings) {
// Normalize boolean and array fields for DB
const normalized = { ...settings };
// Map 'act' from frontend to 'activation' for DB
if (normalized.act !== undefined) {
normalized.activation = normalized.act;
delete normalized.act;
}
// Convert 'on'/'off' to boolean for save_history_ckpt
if (typeof normalized.save_history_ckpt === 'string') {
normalized.save_history_ckpt = normalized.save_history_ckpt === 'on' ? true : false;
}
// Convert comma-separated strings to arrays for input_size, test_size, mosaic_scale, mixup_scale
['input_size', 'test_size', 'mosaic_scale', 'mixup_scale'].forEach(key => {
if (typeof normalized[key] === 'string') {
const arr = normalized[key].split(',').map(v => parseFloat(v.trim()));
normalized[key] = arr.length === 1 ? arr[0] : arr;
}
});
// Find TrainingProjectDetails for this project
const TrainingProjectDetails = require('../models/TrainingProjectDetails.js');
const details = await TrainingProjectDetails.findOne({ where: { project_id: normalized.project_id } });
if (!details) throw new Error('TrainingProjectDetails not found for project_id ' + normalized.project_id);
normalized.project_details_id = details.id;
// Create DB row
const training = await Training.create(normalized);
return training;
}
async function generateYoloxExpFromDb(trainingId) {
// Fetch training row from DB
const training = await Training.findByPk(trainingId);
if (!training) throw new Error('Training not found');
// Template for exp.py
const expTemplate = `#!/usr/bin/env python3\n# Copyright (c) Megvii Inc. All rights reserved.\n\nimport os\nimport random\n\nimport torch\nimport torch.distributed as dist\nimport torch.nn as nn\n\nfrom .base_exp import BaseExp\n\n__all__ = [\"Exp\", \"check_exp_value\"]\n\nclass Exp(BaseExp):\n def __init__(self):\n super().__init__()\n\n # ---------------- model config ---------------- #\n self.num_classes = ${training.num_classes || 80}\n self.depth = ${training.depth || 1.00}\n self.width = ${training.width || 1.00}\n self.act = \"${training.activation || training.act || 'silu'}\"\n\n # ---------------- dataloader config ---------------- #\n self.data_num_workers = ${training.data_num_workers || 4}\n self.input_size = (${Array.isArray(training.input_size) ? training.input_size.join(', ') : '640, 640'})\n self.multiscale_range = ${training.multiscale_range || 5}\n self.data_dir = ${training.data_dir ? `\"${training.data_dir}\"` : 'None'}\n self.train_ann = \"${training.train_ann || 'instances_train2017.json'}\"\n self.val_ann = \"${training.val_ann || 'instances_val2017.json'}\"\n self.test_ann = \"${training.test_ann || 'instances_test2017.json'}\"\n\n # --------------- transform config ----------------- #\n self.mosaic_prob = ${training.mosaic_prob !== undefined ? training.mosaic_prob : 1.0}\n self.mixup_prob = ${training.mixup_prob !== undefined ? training.mixup_prob : 1.0}\n self.hsv_prob = ${training.hsv_prob !== undefined ? training.hsv_prob : 1.0}\n self.flip_prob = ${training.flip_prob !== undefined ? training.flip_prob : 0.5}\n self.degrees = ${training.degrees !== undefined ? training.degrees : 10.0}\n self.translate = ${training.translate !== undefined ? training.translate : 0.1}\n self.mosaic_scale = (${Array.isArray(training.mosaic_scale) ? training.mosaic_scale.join(', ') : '0.1, 2'})\n self.enable_mixup = ${training.enable_mixup !== undefined ? training.enable_mixup : true}\n self.mixup_scale = (${Array.isArray(training.mixup_scale) ? training.mixup_scale.join(', ') : '0.5, 1.5'})\n self.shear = ${training.shear !== undefined ? training.shear : 2.0}\n\n # -------------- training config --------------------- #\n self.warmup_epochs = ${training.warmup_epochs !== undefined ? training.warmup_epochs : 5}\n self.max_epoch = ${training.max_epoch !== undefined ? training.max_epoch : 300}\n self.warmup_lr = ${training.warmup_lr !== undefined ? training.warmup_lr : 0}\n self.min_lr_ratio = ${training.min_lr_ratio !== undefined ? training.min_lr_ratio : 0.05}\n self.basic_lr_per_img = ${training.basic_lr_per_img !== undefined ? training.basic_lr_per_img : 0.01 / 64.0}\n self.scheduler = \"${training.scheduler || 'yoloxwarmcos'}\"\n self.no_aug_epochs = ${training.no_aug_epochs !== undefined ? training.no_aug_epochs : 15}\n self.ema = ${training.ema !== undefined ? training.ema : true}\n self.weight_decay = ${training.weight_decay !== undefined ? training.weight_decay : 5e-4}\n self.momentum = ${training.momentum !== undefined ? training.momentum : 0.9}\n self.print_interval = ${training.print_interval !== undefined ? training.print_interval : 10}\n self.eval_interval = ${training.eval_interval !== undefined ? training.eval_interval : 10}\n self.save_history_ckpt = ${training.save_history_ckpt !== undefined ? training.save_history_ckpt : true}\n self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n # ----------------- testing config ------------------ #\n self.test_size = (${Array.isArray(training.test_size) ? training.test_size.join(', ') : '640, 640'})\n self.test_conf = ${training.test_conf !== undefined ? training.test_conf : 0.01}\n self.nmsthre = ${training.nmsthre !== undefined ? training.nmsthre : 0.65}\n\n # ... rest of the template ...\n\ndef check_exp_value(exp: Exp):\n h, w = exp.input_size\n assert h % 32 == 0 and w % 32 == 0, \"input size must be multiples of 32\"\n`;
// Save to file in output directory
const outDir = path.join(__dirname, '../../', training.project_id ? `project_${training.project_id}/${trainingId}` : 'exp_files');
if (!fs.existsSync(outDir)) fs.mkdirSync(outDir, { recursive: true });
const filePath = path.join(outDir, 'exp.py');
fs.writeFileSync(filePath, expTemplate);
return filePath;
}
module.exports = { pushYoloxExpToDb, generateYoloxExpFromDb };

0
backend/services/push_yolox_exp.py Normal file → Executable file
View File

View File

@@ -1,120 +0,0 @@
const sequelize = require('../database/database.js');
const { Project, Img, Ann } = require('../models');
const { fetchLableStudioProject, fetchProjectIdsAndTitles } = require('./fetch-labelstudio.js');
const updateStatus = { running: false };
async function seedLabelStudio() {
updateStatus.running = true;
console.log('Seeding started');
try {
await sequelize.sync();
const projects = await fetchProjectIdsAndTitles();
for (const project of projects) {
console.log(`Processing project ${project.id} (${project.title})`);
// Upsert project in DB
await Project.upsert({ project_id: project.id, title: project.title });
// Fetch project data (annotations array)
const data = await fetchLableStudioProject(project.id);
if (!Array.isArray(data) || data.length === 0) {
console.log(`No annotation data for project ${project.id}`);
continue;
}
// Remove old images and annotations for this project
const oldImages = await Img.findAll({ where: { project_id: project.id } });
const oldImageIds = oldImages.map(img => img.image_id);
if (oldImageIds.length > 0) {
await Ann.destroy({ where: { image_id: oldImageIds } });
await Img.destroy({ where: { project_id: project.id } });
console.log(`Deleted ${oldImageIds.length} old images and their annotations for project ${project.id}`);
}
// Prepare arrays
const imagesBulk = [];
const annsBulk = [];
for (const ann of data) {
// Extract width/height
let width = null;
let height = null;
if (Array.isArray(ann.label_rectangles) && ann.label_rectangles.length > 0) {
width = ann.label_rectangles[0].original_width;
height = ann.label_rectangles[0].original_height;
} else if (Array.isArray(ann.label) && ann.label.length > 0 && ann.label[0].original_width && ann.label[0].original_height) {
width = ann.label[0].original_width;
height = ann.label[0].original_height;
}
// Only push image and annotations if width and height are valid
if (width && height) {
imagesBulk.push({
project_id: project.id,
image_path: ann.image,
width,
height
});
// Handle multiple annotations per image
if (Array.isArray(ann.label_rectangles)) {
for (const ann_detail of ann.label_rectangles) {
annsBulk.push({
image_path: ann.image,
x: (ann_detail.x * width) / 100,
y: (ann_detail.y * height) / 100,
width: (ann_detail.width * width) / 100,
height: (ann_detail.height * height) / 100,
Label: Array.isArray(ann_detail.rectanglelabels) ? (ann_detail.rectanglelabels[0] || 'unknown') : (ann_detail.rectanglelabels || 'unknown')
});
}
} else if (Array.isArray(ann.label)) {
for (const ann_detail of ann.label) {
annsBulk.push({
image_path: ann.image,
x: (ann_detail.x * width) / 100,
y: (ann_detail.y * height) / 100,
width: (ann_detail.width * width) / 100,
height: (ann_detail.height * height) / 100,
Label: Array.isArray(ann_detail.rectanglelabels) ? (ann_detail.rectanglelabels[0] || 'unknown') : (ann_detail.rectanglelabels || 'unknown')
});
}
}
}
}
// 1) Insert images and get generated IDs
const insertedImages = await Img.bulkCreate(imagesBulk, { returning: true });
// 2) Map image_path -> image_id
const imageMap = {};
for (const img of insertedImages) {
imageMap[img.image_path] = img.image_id;
}
// 3) Assign correct image_id to each annotation
for (const ann of annsBulk) {
ann.image_id = imageMap[ann.image_path];
delete ann.image_path; // cleanup
}
// 4) Insert annotations
await Ann.bulkCreate(annsBulk);
console.log(`Inserted ${imagesBulk.length} images and ${annsBulk.length} annotations for project ${project.id}`);
}
console.log('Seeding done');
return { success: true, message: 'Data inserted successfully!' };
} catch (error) {
console.error('Error inserting data:', error);
return { success: false, message: error.message };
} finally {
updateStatus.running = false;
console.log('updateStatus.running set to false');
}
}
module.exports = { seedLabelStudio, updateStatus };

0
backend/services/seed_label_studio.py Normal file → Executable file
View File

0
backend/services/settings_service.py Normal file → Executable file
View File

View File

@@ -0,0 +1,206 @@
"""
Training Queue Manager
Manages a queue of training jobs and tracks their progress
"""
import threading
import queue
import subprocess
import re
import os
from services.settings_service import get_setting
from models.training import Training
class TrainingQueueManager:
_instance = None
_lock = threading.Lock()
def __new__(cls):
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._initialized = False
return cls._instance
def __init__(self):
if self._initialized:
return
self.queue = queue.Queue()
self.current_training = None
self.current_process = None
self.worker_thread = None
self.running = False
self._initialized = True
# Start the worker thread
self.start_worker()
def start_worker(self):
"""Start the background worker thread"""
if self.worker_thread is None or not self.worker_thread.is_alive():
self.running = True
self.worker_thread = threading.Thread(target=self._process_queue, daemon=True)
self.worker_thread.start()
def add_to_queue(self, training_id, command, cwd):
"""Add a training job to the queue"""
job = {
'training_id': training_id,
'command': command,
'cwd': cwd,
'iteration': 0,
'max_epoch': 300 # Will be updated from training record
}
# Get max_epoch from training record
try:
training = Training.query.get(training_id)
if training:
job['max_epoch'] = training.max_epoch or 300
job['name'] = training.exp_name or f'Training {training_id}'
except:
pass
self.queue.put(job)
print(f'Added training {training_id} to queue. Queue size: {self.queue.qsize()}')
def _process_queue(self):
"""Worker thread that processes the queue"""
while self.running:
try:
# Wait for a job (blocking with timeout)
job = self.queue.get(timeout=1)
print(f'Starting training {job["training_id"]} from queue')
self.current_training = job
# Execute the training command
self._run_training(job)
# Mark as done
self.queue.task_done()
self.current_training = None
self.current_process = None
except queue.Empty:
continue
except Exception as e:
print(f'Error processing training job: {e}')
self.current_training = None
self.current_process = None
def _run_training(self, job):
"""Run a training command and monitor its output"""
try:
import platform
is_windows = platform.system() == 'Windows'
# Start process
self.current_process = subprocess.Popen(
job['command'],
shell=True,
cwd=job['cwd'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1
)
# Monitor output for progress
for line in iter(self.current_process.stdout.readline, ''):
if line:
print(line.strip())
# Parse epoch and iteration from YOLOX output
# Example: "epoch: 3/300, iter: 90/101"
epoch_match = re.search(r'epoch:\s*(\d+)/(\d+)', line, re.IGNORECASE)
iter_match = re.search(r'iter:\s*(\d+)/(\d+)', line, re.IGNORECASE)
if epoch_match:
current_epoch = int(epoch_match.group(1))
total_epochs = int(epoch_match.group(2))
if self.current_training:
self.current_training['current_epoch'] = current_epoch
self.current_training['max_epoch'] = total_epochs
# Debug log
print(f'[PROGRESS] Parsed epoch: {current_epoch}/{total_epochs}')
if iter_match:
current_iter = int(iter_match.group(1))
total_iters = int(iter_match.group(2))
if self.current_training:
self.current_training['current_iter'] = current_iter
self.current_training['total_iters'] = total_iters
# Calculate overall progress percentage
if 'current_epoch' in self.current_training and 'max_epoch' in self.current_training:
epoch_progress = (self.current_training['current_epoch'] - 1) / self.current_training['max_epoch']
iter_progress = current_iter / total_iters / self.current_training['max_epoch']
total_progress = (epoch_progress + iter_progress) * 100
self.current_training['progress'] = round(total_progress, 2)
# Debug log
print(f'[PROGRESS] Epoch {self.current_training["current_epoch"]}/{self.current_training["max_epoch"]}, Iter {current_iter}/{total_iters}, Progress: {self.current_training["progress"]}%')
# Wait for completion
self.current_process.wait()
print(f'Training {job["training_id"]} completed with exit code {self.current_process.returncode}')
except Exception as e:
print(f'Error running training: {e}')
def get_status(self):
"""Get current status of training queue"""
queue_items = []
# Get items from queue without removing them
temp_items = []
while not self.queue.empty():
try:
item = self.queue.get_nowait()
temp_items.append(item)
queue_items.append({
'training_id': item['training_id'],
'name': item.get('name', f'Training {item["training_id"]}'),
'max_epoch': item.get('max_epoch', 300)
})
except queue.Empty:
break
# Put items back
for item in temp_items:
self.queue.put(item)
result = {
'current': None,
'queue': queue_items
}
if self.current_training:
current_epoch = self.current_training.get('current_epoch', 0)
max_epoch = self.current_training.get('max_epoch', 300)
result['current'] = {
'training_id': self.current_training['training_id'],
'name': self.current_training.get('name', f'Training {self.current_training["training_id"]}'),
'epoch': current_epoch, # For backward compatibility
'current_epoch': current_epoch,
'max_epoch': max_epoch,
'current_iter': self.current_training.get('current_iter', 0),
'total_iters': self.current_training.get('total_iters', 0),
'progress': self.current_training.get('progress', 0.0),
'iteration': current_epoch # For backward compatibility
}
return result
def stop(self):
"""Stop the worker thread"""
self.running = False
if self.current_process:
try:
self.current_process.terminate()
except:
pass
# Global instance
training_queue = TrainingQueueManager()

View File

@@ -0,0 +1,244 @@
"""
Validate dataset for training - check for problematic images and annotations
"""
import os
import json
from PIL import Image
import cv2
def validate_coco_json(json_path, data_dir):
"""
Validate a COCO JSON file and check all images
Args:
json_path: Path to COCO JSON file
data_dir: Directory where images are located
Returns:
dict with validation results
"""
print(f"\n{'='*60}")
print(f"Validating: {json_path}")
print(f"{'='*60}\n")
issues = {
'missing_images': [],
'corrupted_images': [],
'zero_dimension_images': [],
'invalid_annotations': [],
'zero_area_boxes': []
}
try:
with open(json_path, 'r') as f:
coco_data = json.load(f)
except Exception as e:
print(f"❌ Failed to load JSON: {e}")
return issues
images = coco_data.get('images', [])
annotations = coco_data.get('annotations', [])
print(f"📊 Dataset Stats:")
print(f" Images: {len(images)}")
print(f" Annotations: {len(annotations)}")
print(f" Categories: {len(coco_data.get('categories', []))}")
print()
# Validate images
print("🔍 Validating images...")
for idx, img_info in enumerate(images):
img_id = img_info.get('id')
file_name = img_info.get('file_name', '')
width = img_info.get('width', 0)
height = img_info.get('height', 0)
# Check if image file exists
# Try to construct the full path
if os.path.isabs(file_name):
img_path = file_name
else:
img_path = os.path.join(data_dir, file_name)
if not os.path.exists(img_path):
issues['missing_images'].append({
'id': img_id,
'file_name': file_name,
'expected_path': img_path
})
continue
# Check if image can be loaded
try:
# Try with PIL
with Image.open(img_path) as pil_img:
pil_width, pil_height = pil_img.size
# Check if dimensions match JSON
if pil_width != width or pil_height != height:
print(f"⚠️ Image {img_id}: Dimension mismatch - JSON: {width}x{height}, Actual: {pil_width}x{pil_height}")
# Check for zero dimensions
if pil_width == 0 or pil_height == 0:
issues['zero_dimension_images'].append({
'id': img_id,
'file_name': file_name,
'dimensions': f"{pil_width}x{pil_height}"
})
except Exception as e:
issues['corrupted_images'].append({
'id': img_id,
'file_name': file_name,
'error': str(e)
})
# Progress indicator
if (idx + 1) % 100 == 0:
print(f" Checked {idx + 1}/{len(images)} images...")
print(f"✅ Image validation complete\n")
# Validate annotations
print("🔍 Validating annotations...")
for idx, ann in enumerate(annotations):
ann_id = ann.get('id')
img_id = ann.get('image_id')
bbox = ann.get('bbox', [])
if len(bbox) != 4:
issues['invalid_annotations'].append({
'id': ann_id,
'image_id': img_id,
'reason': f'Invalid bbox length: {len(bbox)}'
})
continue
x, y, w, h = bbox
# Check for zero or negative dimensions
if w <= 0 or h <= 0:
issues['zero_area_boxes'].append({
'id': ann_id,
'image_id': img_id,
'bbox': bbox,
'reason': f'Zero or negative dimensions: w={w}, h={h}'
})
# Check for extremely small boxes (potential issue with mixup)
if w < 1 or h < 1:
issues['zero_area_boxes'].append({
'id': ann_id,
'image_id': img_id,
'bbox': bbox,
'reason': f'Extremely small box: w={w}, h={h}'
})
# Progress indicator
if (idx + 1) % 1000 == 0:
print(f" Checked {idx + 1}/{len(annotations)} annotations...")
print(f"✅ Annotation validation complete\n")
# Print summary
print(f"\n{'='*60}")
print("VALIDATION SUMMARY")
print(f"{'='*60}\n")
total_issues = sum(len(v) for v in issues.values())
if total_issues == 0:
print("✅ No issues found! Dataset is ready for training.")
else:
print(f"⚠️ Found {total_issues} total issues:\n")
if issues['missing_images']:
print(f" ❌ Missing images: {len(issues['missing_images'])}")
for item in issues['missing_images'][:5]: # Show first 5
print(f" - {item['file_name']}")
if len(issues['missing_images']) > 5:
print(f" ... and {len(issues['missing_images']) - 5} more")
if issues['corrupted_images']:
print(f" ❌ Corrupted images: {len(issues['corrupted_images'])}")
for item in issues['corrupted_images'][:5]:
print(f" - {item['file_name']}: {item['error']}")
if len(issues['corrupted_images']) > 5:
print(f" ... and {len(issues['corrupted_images']) - 5} more")
if issues['zero_dimension_images']:
print(f" ❌ Zero dimension images: {len(issues['zero_dimension_images'])}")
for item in issues['zero_dimension_images'][:5]:
print(f" - {item['file_name']}: {item['dimensions']}")
if len(issues['zero_dimension_images']) > 5:
print(f" ... and {len(issues['zero_dimension_images']) - 5} more")
if issues['invalid_annotations']:
print(f" ❌ Invalid annotations: {len(issues['invalid_annotations'])}")
for item in issues['invalid_annotations'][:5]:
print(f" - Ann ID {item['id']}: {item['reason']}")
if len(issues['invalid_annotations']) > 5:
print(f" ... and {len(issues['invalid_annotations']) - 5} more")
if issues['zero_area_boxes']:
print(f" ⚠️ Zero/tiny area boxes: {len(issues['zero_area_boxes'])}")
print(f" These may cause issues with mixup augmentation!")
for item in issues['zero_area_boxes'][:5]:
print(f" - Ann ID {item['id']}, bbox: {item['bbox']}")
if len(issues['zero_area_boxes']) > 5:
print(f" ... and {len(issues['zero_area_boxes']) - 5} more")
print()
return issues
def validate_training_dataset(training_id):
"""
Validate all COCO JSON files for a training
Args:
training_id: The training ID to validate
"""
from models.training import Training
from models.TrainingProject import TrainingProject
from services.settings_service import get_setting
training = Training.query.get(training_id)
if not training:
print(f"❌ Training {training_id} not found")
return
# Get paths
from models.TrainingProjectDetails import TrainingProjectDetails
details = TrainingProjectDetails.query.get(training.project_details_id)
training_project = TrainingProject.query.get(details.project_id)
project_name = training_project.title.replace(' ', '_') if training_project else f'project_{details.project_id}'
training_folder_name = f"{training.exp_name or training.training_name or 'training'}_{training_id}"
training_folder_name = training_folder_name.replace(' ', '_')
output_base_path = get_setting('yolox_output_path', './backend')
data_dir = get_setting('yolox_data_dir', '/home/kitraining/To_Annotate/')
annotations_dir = os.path.join(output_base_path, project_name, training_folder_name, 'annotations')
# Validate each split
splits = ['train', 'valid', 'test']
all_issues = {}
for split in splits:
json_file = os.path.join(annotations_dir, f'coco_project_{training_id}_{split}.json')
if os.path.exists(json_file):
all_issues[split] = validate_coco_json(json_file, data_dir)
else:
print(f"⚠️ JSON file not found: {json_file}")
return all_issues
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
training_id = int(sys.argv[1])
validate_training_dataset(training_id)
else:
print("Usage: python validate_dataset.py <training_id>")

0
backend/start.py Normal file → Executable file
View File

0
backend/test/7/exp.py Normal file → Executable file
View File

0
documentation/Projektdoku.pdf Normal file → Executable file
View File

0
documentation/Projektdokumentation.md Normal file → Executable file
View File

192
edit-training.html Normal file → Executable file
View File

@@ -202,6 +202,14 @@
<label id="project-title-label" <label id="project-title-label"
style="display: block; text-align: left; font-weight: bold; font-size: x-large;">title</label> style="display: block; text-align: left; font-weight: bold; font-size: x-large;">title</label>
<div class="button-row"> <div class="button-row">
<!-- Training Notification Bell -->
<button id="training-bell" onclick="toggleTrainingModal()" class="button" title="Training Status"
style="padding: 8px 16px; margin-right: 10px; position: relative; background: #999;">
🔔
<span id="bell-badge" style="display: none; position: absolute; top: -5px; right: -5px; background: #ff4d4f;
color: white; border-radius: 50%; width: 20px; height: 20px; font-size: 12px; line-height: 20px;
text-align: center; font-weight: bold;">0</span>
</button>
<button id="Add Training Project" onclick="window.location.href='/add-project.html'" <button id="Add Training Project" onclick="window.location.href='/add-project.html'"
class="button-red">Add Training Project</button> class="button-red">Add Training Project</button>
<button id="seed-db-btn" class="button"> <button id="seed-db-btn" class="button">
@@ -553,7 +561,7 @@ document.getElementById('parameters-form').addEventListener('submit', async func
} }
// Add project id if available // Add project id if available
const urlParams = new URLSearchParams(window.location.search); const urlParams = new URLSearchParams(window.location.search);
const projectId = urlParams.get('id'); const projectId = urlParams.get('project_id') || urlParams.get('id'); // Support both parameter names
if (projectId) formData.append('project_id', projectId); if (projectId) formData.append('project_id', projectId);
// Send to backend // Send to backend
try { try {
@@ -640,4 +648,186 @@ document.getElementById('parameters-form').addEventListener('submit', async func
</div> </div>
<script src="js/settings.js"></script> <script src="js/settings.js"></script>
<script>
// Handle readonly mode and load training data
window.addEventListener('DOMContentLoaded', function() {
const urlParams = new URLSearchParams(window.location.search);
const trainingId = urlParams.get('training_id');
const isReadonly = urlParams.get('readonly') === 'true';
if (isReadonly) {
// Update page title
const titleLabel = document.querySelector('#project-title-label');
if (titleLabel) {
titleLabel.textContent = 'View Training Details (Read-Only)';
}
// Disable all form inputs
document.querySelectorAll('input, select, textarea, button[type="submit"]').forEach(el => {
el.disabled = true;
el.style.opacity = '0.6';
el.style.cursor = 'not-allowed';
});
// Hide submit buttons
document.querySelectorAll('button[type="submit"]').forEach(btn => {
btn.style.display = 'none';
});
// Re-enable navigation buttons (back, etc.)
document.querySelectorAll('button:not([type="submit"])').forEach(btn => {
if (btn.textContent.includes('Back') || btn.onclick) {
btn.disabled = false;
btn.style.opacity = '1';
btn.style.cursor = 'pointer';
}
});
}
// Load training data if training_id is present
if (trainingId) {
fetch(`/api/trainings/${trainingId}`)
.then(res => res.json())
.then(training => {
// Populate form fields with training data
Object.keys(training).forEach(key => {
const input = document.querySelector(`[name="${key}"]`);
if (input) {
if (input.type === 'checkbox') {
input.checked = training[key];
} else {
input.value = training[key] || '';
}
}
});
// Handle special fields
if (training.train) document.getElementById('train-slider').value = training.train;
if (training.valid) document.getElementById('valid-slider').value = training.valid;
if (training.test) document.getElementById('test-slider').value = training.test;
syncSplitSliders('train');
})
.catch(err => {
console.error('Failed to load training data:', err);
});
}
});
</script>
<!-- Training Status Modal -->
<div id="training-status-modal" class="modal" style="display: none;">
<div class="modal-content" style="max-width: 700px; max-height: 90vh; overflow-y: auto;">
<div class="modal-header">
<h2>Training Status</h2>
<button class="close-btn" onclick="toggleTrainingModal()">&times;</button>
</div>
<div class="modal-body">
<div class="settings-section" id="current-training-section" style="display: none;">
<h3 style="color: #009eac;">Current Training</h3>
<div id="current-training-info" style="background: #eaf7fa; padding: 16px; border-radius: 8px; margin-bottom: 16px;">
</div>
</div>
<div class="settings-section" id="queue-section" style="display: none;">
<h3 style="color: #666;">Queue (<span id="queue-count">0</span>)</h3>
<div id="queue-list" style="display: flex; flex-direction: column; gap: 12px;">
</div>
</div>
<div id="no-trainings-msg" style="text-align: center; padding: 32px; color: #666;">
No trainings running or queued.
</div>
</div>
</div>
</div>
<script>
let trainingStatusPoller = null;
function toggleTrainingModal() {
const modal = document.getElementById('training-status-modal');
if (modal.style.display === 'none') {
modal.style.display = 'flex';
updateTrainingStatus();
} else {
modal.style.display = 'none';
}
}
function updateTrainingStatus() {
fetch('/api/training-status')
.then(res => res.json())
.then(data => {
const bell = document.getElementById('training-bell');
const badge = document.getElementById('bell-badge');
const currentSection = document.getElementById('current-training-section');
const queueSection = document.getElementById('queue-section');
const noTrainingsMsg = document.getElementById('no-trainings-msg');
const totalCount = (data.current ? 1 : 0) + data.queue.length;
if (totalCount > 0) {
bell.style.background = '#009eac';
badge.style.display = 'block';
badge.textContent = totalCount;
} else {
bell.style.background = '#999';
badge.style.display = 'none';
}
if (data.current) {
currentSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
const percentage = Math.round((data.current.iteration / data.current.max_epoch) * 100);
document.getElementById('current-training-info').innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px;">
<strong>${data.current.name || 'Training'}</strong>
<span style="font-weight: bold; color: #009eac;">${percentage}%</span>
</div>
<div style="background: #ddd; border-radius: 4px; height: 24px; overflow: hidden; margin-bottom: 8px;">
<div style="background: #009eac; height: 100%; width: ${percentage}%; transition: width 0.3s;"></div>
</div>
<div style="font-size: 14px; color: #666;">
Epoch ${data.current.iteration} / ${data.current.max_epoch}
</div>
`;
} else {
currentSection.style.display = 'none';
}
if (data.queue.length > 0) {
queueSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
document.getElementById('queue-count').textContent = data.queue.length;
document.getElementById('queue-list').innerHTML = data.queue.map((t, idx) => `
<div style="background: #f5f5f5; padding: 12px; border-radius: 8px; border-left: 4px solid #009eac;">
<strong>#${idx + 1}: ${t.name || 'Training'}</strong>
<div style="font-size: 13px; color: #666; margin-top: 4px;">
${t.max_epoch} epochs • Waiting...
</div>
</div>
`).join('');
} else {
queueSection.style.display = 'none';
}
if (totalCount === 0) {
noTrainingsMsg.style.display = 'block';
}
})
.catch(err => console.error('Failed to fetch training status:', err));
}
window.addEventListener('DOMContentLoaded', function() {
updateTrainingStatus();
trainingStatusPoller = setInterval(updateTrainingStatus, 5000);
});
window.addEventListener('beforeunload', function() {
if (trainingStatusPoller) clearInterval(trainingStatusPoller);
});
</script>
</body></html> </body></html>

0
globals.css Normal file → Executable file
View File

134
index.html Normal file → Executable file
View File

@@ -28,6 +28,14 @@
<icon class="header-icon" onclick="window.location.href='/index.html'" , onmouseover="" style="cursor: pointer;" <icon class="header-icon" onclick="window.location.href='/index.html'" , onmouseover="" style="cursor: pointer;"
src="./media/logo.png" alt="Logo"></icon> src="./media/logo.png" alt="Logo"></icon>
<div class="button-row"> <div class="button-row">
<!-- Training Notification Bell -->
<button id="training-bell" onclick="toggleTrainingModal()" class="button" title="Training Status"
style="padding: 8px 16px; margin-right: 10px; position: relative; background: #999;">
🔔
<span id="bell-badge" style="display: none; position: absolute; top: -5px; right: -5px; background: #ff4d4f;
color: white; border-radius: 50%; width: 20px; height: 20px; font-size: 12px; line-height: 20px;
text-align: center; font-weight: bold;">0</span>
</button>
<button id="Add Training Project" onclick="window.location.href='/add-project.html'" class="button-red">Add <button id="Add Training Project" onclick="window.location.href='/add-project.html'" class="button-red">Add
Training Project</button> Training Project</button>
<button id="seed-db-btn" class="button"> <button id="seed-db-btn" class="button">
@@ -159,7 +167,133 @@
</div> </div>
</div> </div>
<!-- Training Status Modal -->
<div id="training-status-modal" class="modal" style="display: none;">
<div class="modal-content" style="max-width: 700px; max-height: 90vh; overflow-y: auto;">
<div class="modal-header">
<h2>Training Status</h2>
<button class="close-btn" onclick="toggleTrainingModal()">&times;</button>
</div>
<div class="modal-body">
<!-- Current Training -->
<div class="settings-section" id="current-training-section" style="display: none;">
<h3 style="color: #009eac;">Current Training</h3>
<div id="current-training-info" style="background: #eaf7fa; padding: 16px; border-radius: 8px; margin-bottom: 16px;">
<!-- Populated by JS -->
</div>
</div>
<!-- Queued Trainings -->
<div class="settings-section" id="queue-section" style="display: none;">
<h3 style="color: #666;">Queue (<span id="queue-count">0</span>)</h3>
<div id="queue-list" style="display: flex; flex-direction: column; gap: 12px;">
<!-- Populated by JS -->
</div>
</div>
<!-- No trainings message -->
<div id="no-trainings-msg" style="text-align: center; padding: 32px; color: #666;">
No trainings running or queued.
</div>
</div>
</div>
</div>
<script src="js/settings.js"></script> <script src="js/settings.js"></script>
<script>
// Training status polling
let trainingStatusPoller = null;
function toggleTrainingModal() {
const modal = document.getElementById('training-status-modal');
if (modal.style.display === 'none') {
modal.style.display = 'flex';
updateTrainingStatus(); // Immediate update
} else {
modal.style.display = 'none';
}
}
function updateTrainingStatus() {
fetch('/api/training-status')
.then(res => res.json())
.then(data => {
const bell = document.getElementById('training-bell');
const badge = document.getElementById('bell-badge');
const currentSection = document.getElementById('current-training-section');
const queueSection = document.getElementById('queue-section');
const noTrainingsMsg = document.getElementById('no-trainings-msg');
const totalCount = (data.current ? 1 : 0) + data.queue.length;
// Update bell appearance
if (totalCount > 0) {
bell.style.background = '#009eac';
badge.style.display = 'block';
badge.textContent = totalCount;
} else {
bell.style.background = '#999';
badge.style.display = 'none';
}
// Update modal content
if (data.current) {
currentSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
const percentage = Math.round((data.current.iteration / data.current.max_epoch) * 100);
document.getElementById('current-training-info').innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px;">
<strong>${data.current.name || 'Training'}</strong>
<span style="font-weight: bold; color: #009eac;">${percentage}%</span>
</div>
<div style="background: #ddd; border-radius: 4px; height: 24px; overflow: hidden; margin-bottom: 8px;">
<div style="background: #009eac; height: 100%; width: ${percentage}%; transition: width 0.3s;"></div>
</div>
<div style="font-size: 14px; color: #666;">
Epoch ${data.current.iteration} / ${data.current.max_epoch}
</div>
`;
} else {
currentSection.style.display = 'none';
}
if (data.queue.length > 0) {
queueSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
document.getElementById('queue-count').textContent = data.queue.length;
document.getElementById('queue-list').innerHTML = data.queue.map((t, idx) => `
<div style="background: #f5f5f5; padding: 12px; border-radius: 8px; border-left: 4px solid #009eac;">
<strong>#${idx + 1}: ${t.name || 'Training'}</strong>
<div style="font-size: 13px; color: #666; margin-top: 4px;">
${t.max_epoch} epochs • Waiting...
</div>
</div>
`).join('');
} else {
queueSection.style.display = 'none';
}
if (totalCount === 0) {
noTrainingsMsg.style.display = 'block';
}
})
.catch(err => console.error('Failed to fetch training status:', err));
}
// Poll every 5 seconds
window.addEventListener('DOMContentLoaded', function() {
updateTrainingStatus();
trainingStatusPoller = setInterval(updateTrainingStatus, 5000);
});
// Stop polling when page unloads
window.addEventListener('beforeunload', function() {
if (trainingStatusPoller) clearInterval(trainingStatusPoller);
});
</script>
</body> </body>
</html> </html>

0
js/add-class.js Normal file → Executable file
View File

0
js/add-image.js Normal file → Executable file
View File

0
js/dashboard-label-studio.js Normal file → Executable file
View File

0
js/dashboard.js Normal file → Executable file
View File

0
js/overview-training.js Normal file → Executable file
View File

0
js/settings.js Normal file → Executable file
View File

0
js/setup-training-project.js Normal file → Executable file
View File

0
js/start-training.js Normal file → Executable file
View File

0
js/storage.js Normal file → Executable file
View File

0
media/logo.png Normal file → Executable file
View File

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 11 KiB

154
overview-training.html Normal file → Executable file
View File

@@ -30,6 +30,14 @@
<label id="project-title-label" <label id="project-title-label"
style="display: block; text-align: left; font-weight: bold; font-size: x-large;">Project</label> style="display: block; text-align: left; font-weight: bold; font-size: x-large;">Project</label>
<div class="button-row"> <div class="button-row">
<!-- Training Notification Bell -->
<button id="training-bell" onclick="toggleTrainingModal()" class="button" title="Training Status"
style="padding: 8px 16px; margin-right: 10px; position: relative; background: #999;">
🔔
<span id="bell-badge" style="display: none; position: absolute; top: -5px; right: -5px; background: #ff4d4f;
color: white; border-radius: 50%; width: 20px; height: 20px; font-size: 12px; line-height: 20px;
text-align: center; font-weight: bold;">0</span>
</button>
<button id="Add Training Project" onclick="window.location.href='/add-project.html'" class="button-red">Add <button id="Add Training Project" onclick="window.location.href='/add-project.html'" class="button-red">Add
Training Project</button> Training Project</button>
<button id="seed-db-btn" class="button"> <button id="seed-db-btn" class="button">
@@ -44,10 +52,7 @@
</div> </div>
</button> </button>
</button>
<button id="setup-details" class="button">
Show Details
</button>
<script> <script>
document.getElementById('seed-db-btn').addEventListener('click', function () { document.getElementById('seed-db-btn').addEventListener('click', function () {
@@ -156,14 +161,15 @@
}; };
btnDiv.appendChild(startBtn); btnDiv.appendChild(startBtn);
// View Log button // View Training Details button
const logBtn = document.createElement('button'); const detailsBtn = document.createElement('button');
logBtn.textContent = 'View Training Log'; detailsBtn.textContent = 'View Training Details';
logBtn.style = 'background:#666;color:white;border:none;border-radius:6px;padding:6px 12px;cursor:pointer;'; detailsBtn.style = 'background:#666;color:white;border:none;border-radius:6px;padding:6px 12px;cursor:pointer;';
logBtn.onclick = function() { detailsBtn.onclick = function() {
showLogModal(training.id); // Navigate to edit-training page in read-only mode
window.location.href = `/edit-training.html?training_id=${training.id}&readonly=true`;
}; };
btnDiv.appendChild(logBtn); btnDiv.appendChild(detailsBtn);
// Remove button // Remove button
const removeBtn = document.createElement('button'); const removeBtn = document.createElement('button');
@@ -319,7 +325,133 @@
</div> </div>
</div> </div>
<!-- Training Status Modal -->
<div id="training-status-modal" class="modal" style="display: none;">
<div class="modal-content" style="max-width: 700px; max-height: 90vh; overflow-y: auto;">
<div class="modal-header">
<h2>Training Status</h2>
<button class="close-btn" onclick="toggleTrainingModal()">&times;</button>
</div>
<div class="modal-body">
<!-- Current Training -->
<div class="settings-section" id="current-training-section" style="display: none;">
<h3 style="color: #009eac;">Current Training</h3>
<div id="current-training-info" style="background: #eaf7fa; padding: 16px; border-radius: 8px; margin-bottom: 16px;">
<!-- Populated by JS -->
</div>
</div>
<!-- Queued Trainings -->
<div class="settings-section" id="queue-section" style="display: none;">
<h3 style="color: #666;">Queue (<span id="queue-count">0</span>)</h3>
<div id="queue-list" style="display: flex; flex-direction: column; gap: 12px;">
<!-- Populated by JS -->
</div>
</div>
<!-- No trainings message -->
<div id="no-trainings-msg" style="text-align: center; padding: 32px; color: #666;">
No trainings running or queued.
</div>
</div>
</div>
</div>
<script src="js/settings.js"></script> <script src="js/settings.js"></script>
<script>
// Training status polling
let trainingStatusPoller = null;
function toggleTrainingModal() {
const modal = document.getElementById('training-status-modal');
if (modal.style.display === 'none') {
modal.style.display = 'flex';
updateTrainingStatus(); // Immediate update
} else {
modal.style.display = 'none';
}
}
function updateTrainingStatus() {
fetch('/api/training-status')
.then(res => res.json())
.then(data => {
const bell = document.getElementById('training-bell');
const badge = document.getElementById('bell-badge');
const currentSection = document.getElementById('current-training-section');
const queueSection = document.getElementById('queue-section');
const noTrainingsMsg = document.getElementById('no-trainings-msg');
const totalCount = (data.current ? 1 : 0) + data.queue.length;
// Update bell appearance
if (totalCount > 0) {
bell.style.background = '#009eac';
badge.style.display = 'block';
badge.textContent = totalCount;
} else {
bell.style.background = '#999';
badge.style.display = 'none';
}
// Update modal content
if (data.current) {
currentSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
const percentage = Math.round((data.current.iteration / data.current.max_epoch) * 100);
document.getElementById('current-training-info').innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px;">
<strong>${data.current.name || 'Training'}</strong>
<span style="font-weight: bold; color: #009eac;">${percentage}%</span>
</div>
<div style="background: #ddd; border-radius: 4px; height: 24px; overflow: hidden; margin-bottom: 8px;">
<div style="background: #009eac; height: 100%; width: ${percentage}%; transition: width 0.3s;"></div>
</div>
<div style="font-size: 14px; color: #666;">
Epoch ${data.current.iteration} / ${data.current.max_epoch}
</div>
`;
} else {
currentSection.style.display = 'none';
}
if (data.queue.length > 0) {
queueSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
document.getElementById('queue-count').textContent = data.queue.length;
document.getElementById('queue-list').innerHTML = data.queue.map((t, idx) => `
<div style="background: #f5f5f5; padding: 12px; border-radius: 8px; border-left: 4px solid #009eac;">
<strong>#${idx + 1}: ${t.name || 'Training'}</strong>
<div style="font-size: 13px; color: #666; margin-top: 4px;">
${t.max_epoch} epochs • Waiting...
</div>
</div>
`).join('');
} else {
queueSection.style.display = 'none';
}
if (totalCount === 0) {
noTrainingsMsg.style.display = 'block';
}
})
.catch(err => console.error('Failed to fetch training status:', err));
}
// Poll every 5 seconds
window.addEventListener('DOMContentLoaded', function() {
updateTrainingStatus();
trainingStatusPoller = setInterval(updateTrainingStatus, 5000);
});
// Stop polling when page unloads
window.addEventListener('beforeunload', function() {
if (trainingStatusPoller) clearInterval(trainingStatusPoller);
});
</script>
</body> </body>
</html> </html>

124
project-details.html Normal file → Executable file
View File

@@ -32,6 +32,14 @@
style="cursor: pointer;" src="./media/logo.png" alt="Logo"></icon> style="cursor: pointer;" src="./media/logo.png" alt="Logo"></icon>
<label id="project-title-label" style="display: block; text-align: left; font-weight: bold; font-size: x-large;">title</label> <label id="project-title-label" style="display: block; text-align: left; font-weight: bold; font-size: x-large;">title</label>
<div class="button-row"> <div class="button-row">
<!-- Training Notification Bell -->
<button id="training-bell" onclick="toggleTrainingModal()" class="button" title="Training Status"
style="padding: 8px 16px; margin-right: 10px; position: relative; background: #999;">
🔔
<span id="bell-badge" style="display: none; position: absolute; top: -5px; right: -5px; background: #ff4d4f;
color: white; border-radius: 50%; width: 20px; height: 20px; font-size: 12px; line-height: 20px;
text-align: center; font-weight: bold;">0</span>
</button>
<button id="Add Training Project" onclick="window.location.href='/add-project.html'" <button id="Add Training Project" onclick="window.location.href='/add-project.html'"
class="button-red">Add Training Project</button> class="button-red">Add Training Project</button>
<button id="seed-db-btn" class="button"> <button id="seed-db-btn" class="button">
@@ -177,7 +185,123 @@
</div> </div>
</div> </div>
<!-- Training Status Modal -->
<div id="training-status-modal" class="modal" style="display: none;">
<div class="modal-content" style="max-width: 700px; max-height: 90vh; overflow-y: auto;">
<div class="modal-header">
<h2>Training Status</h2>
<button class="close-btn" onclick="toggleTrainingModal()">&times;</button>
</div>
<div class="modal-body">
<div class="settings-section" id="current-training-section" style="display: none;">
<h3 style="color: #009eac;">Current Training</h3>
<div id="current-training-info" style="background: #eaf7fa; padding: 16px; border-radius: 8px; margin-bottom: 16px;">
</div>
</div>
<div class="settings-section" id="queue-section" style="display: none;">
<h3 style="color: #666;">Queue (<span id="queue-count">0</span>)</h3>
<div id="queue-list" style="display: flex; flex-direction: column; gap: 12px;">
</div>
</div>
<div id="no-trainings-msg" style="text-align: center; padding: 32px; color: #666;">
No trainings running or queued.
</div>
</div>
</div>
</div>
<script src="js/settings.js"></script> <script src="js/settings.js"></script>
<script>
let trainingStatusPoller = null;
function toggleTrainingModal() {
const modal = document.getElementById('training-status-modal');
if (modal.style.display === 'none') {
modal.style.display = 'flex';
updateTrainingStatus();
} else {
modal.style.display = 'none';
}
}
function updateTrainingStatus() {
fetch('/api/training-status')
.then(res => res.json())
.then(data => {
const bell = document.getElementById('training-bell');
const badge = document.getElementById('bell-badge');
const currentSection = document.getElementById('current-training-section');
const queueSection = document.getElementById('queue-section');
const noTrainingsMsg = document.getElementById('no-trainings-msg');
const totalCount = (data.current ? 1 : 0) + data.queue.length;
if (totalCount > 0) {
bell.style.background = '#009eac';
badge.style.display = 'block';
badge.textContent = totalCount;
} else {
bell.style.background = '#999';
badge.style.display = 'none';
}
if (data.current) {
currentSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
const percentage = Math.round((data.current.iteration / data.current.max_epoch) * 100);
document.getElementById('current-training-info').innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px;">
<strong>${data.current.name || 'Training'}</strong>
<span style="font-weight: bold; color: #009eac;">${percentage}%</span>
</div>
<div style="background: #ddd; border-radius: 4px; height: 24px; overflow: hidden; margin-bottom: 8px;">
<div style="background: #009eac; height: 100%; width: ${percentage}%; transition: width 0.3s;"></div>
</div>
<div style="font-size: 14px; color: #666;">
Epoch ${data.current.iteration} / ${data.current.max_epoch}
</div>
`;
} else {
currentSection.style.display = 'none';
}
if (data.queue.length > 0) {
queueSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
document.getElementById('queue-count').textContent = data.queue.length;
document.getElementById('queue-list').innerHTML = data.queue.map((t, idx) => `
<div style="background: #f5f5f5; padding: 12px; border-radius: 8px; border-left: 4px solid #009eac;">
<strong>#${idx + 1}: ${t.name || 'Training'}</strong>
<div style="font-size: 13px; color: #666; margin-top: 4px;">
${t.max_epoch} epochs • Waiting...
</div>
</div>
`).join('');
} else {
queueSection.style.display = 'none';
}
if (totalCount === 0) {
noTrainingsMsg.style.display = 'block';
}
})
.catch(err => console.error('Failed to fetch training status:', err));
}
window.addEventListener('DOMContentLoaded', function() {
updateTrainingStatus();
trainingStatusPoller = setInterval(updateTrainingStatus, 5000);
});
window.addEventListener('beforeunload', function() {
if (trainingStatusPoller) clearInterval(trainingStatusPoller);
});
</script>
</body> </body>
</html> </html>

0
settings.html Normal file → Executable file
View File

130
setup-training-project.html Normal file → Executable file
View File

@@ -29,6 +29,14 @@
src="./media/logo.png" alt="Logo"></icon> src="./media/logo.png" alt="Logo"></icon>
<label id="project-title-label" style="display: block; text-align: left; font-weight: bold; font-size: x-large;">title</label> <label id="project-title-label" style="display: block; text-align: left; font-weight: bold; font-size: x-large;">title</label>
<div class="button-row"> <div class="button-row">
<!-- Training Notification Bell -->
<button id="training-bell" onclick="toggleTrainingModal()" class="button" title="Training Status"
style="padding: 8px 16px; margin-right: 10px; position: relative; background: #999;">
🔔
<span id="bell-badge" style="display: none; position: absolute; top: -5px; right: -5px; background: #ff4d4f;
color: white; border-radius: 50%; width: 20px; height: 20px; font-size: 12px; line-height: 20px;
text-align: center; font-weight: bold;">0</span>
</button>
<button id="Add Training Project" onclick="window.location.href='/add-project.html'" class="button-red">Add <button id="Add Training Project" onclick="window.location.href='/add-project.html'" class="button-red">Add
Training Project</button> Training Project</button>
<button id="seed-db-btn" class="button"> <button id="seed-db-btn" class="button">
@@ -170,7 +178,129 @@
</div> </div>
</div> </div>
<!-- Training Status Modal -->
<div id="training-status-modal" class="modal" style="display: none;">
<div class="modal-content" style="max-width: 700px; max-height: 90vh; overflow-y: auto;">
<div class="modal-header">
<h2>Training Status</h2>
<button class="close-btn" onclick="toggleTrainingModal()">&times;</button>
</div>
<div class="modal-body">
<!-- Current Training -->
<div class="settings-section" id="current-training-section" style="display: none;">
<h3 style="color: #009eac;">Current Training</h3>
<div id="current-training-info" style="background: #eaf7fa; padding: 16px; border-radius: 8px; margin-bottom: 16px;">
<!-- Populated by JS -->
</div>
</div>
<!-- Queued Trainings -->
<div class="settings-section" id="queue-section" style="display: none;">
<h3 style="color: #666;">Queue (<span id="queue-count">0</span>)</h3>
<div id="queue-list" style="display: flex; flex-direction: column; gap: 12px;">
<!-- Populated by JS -->
</div>
</div>
<!-- No trainings message -->
<div id="no-trainings-msg" style="text-align: center; padding: 32px; color: #666;">
No trainings running or queued.
</div>
</div>
</div>
</div>
<script src="js/settings.js"></script> <script src="js/settings.js"></script>
<script>
// Training status polling
let trainingStatusPoller = null;
function toggleTrainingModal() {
const modal = document.getElementById('training-status-modal');
if (modal.style.display === 'none') {
modal.style.display = 'flex';
updateTrainingStatus();
} else {
modal.style.display = 'none';
}
}
function updateTrainingStatus() {
fetch('/api/training-status')
.then(res => res.json())
.then(data => {
const bell = document.getElementById('training-bell');
const badge = document.getElementById('bell-badge');
const currentSection = document.getElementById('current-training-section');
const queueSection = document.getElementById('queue-section');
const noTrainingsMsg = document.getElementById('no-trainings-msg');
const totalCount = (data.current ? 1 : 0) + data.queue.length;
if (totalCount > 0) {
bell.style.background = '#009eac';
badge.style.display = 'block';
badge.textContent = totalCount;
} else {
bell.style.background = '#999';
badge.style.display = 'none';
}
if (data.current) {
currentSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
const percentage = Math.round((data.current.iteration / data.current.max_epoch) * 100);
document.getElementById('current-training-info').innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px;">
<strong>${data.current.name || 'Training'}</strong>
<span style="font-weight: bold; color: #009eac;">${percentage}%</span>
</div>
<div style="background: #ddd; border-radius: 4px; height: 24px; overflow: hidden; margin-bottom: 8px;">
<div style="background: #009eac; height: 100%; width: ${percentage}%; transition: width 0.3s;"></div>
</div>
<div style="font-size: 14px; color: #666;">
Epoch ${data.current.iteration} / ${data.current.max_epoch}
</div>
`;
} else {
currentSection.style.display = 'none';
}
if (data.queue.length > 0) {
queueSection.style.display = 'block';
noTrainingsMsg.style.display = 'none';
document.getElementById('queue-count').textContent = data.queue.length;
document.getElementById('queue-list').innerHTML = data.queue.map((t, idx) => `
<div style="background: #f5f5f5; padding: 12px; border-radius: 8px; border-left: 4px solid #009eac;">
<strong>#${idx + 1}: ${t.name || 'Training'}</strong>
<div style="font-size: 13px; color: #666; margin-top: 4px;">
${t.max_epoch} epochs • Waiting...
</div>
</div>
`).join('');
} else {
queueSection.style.display = 'none';
}
if (totalCount === 0) {
noTrainingsMsg.style.display = 'block';
}
})
.catch(err => console.error('Failed to fetch training status:', err));
}
window.addEventListener('DOMContentLoaded', function() {
updateTrainingStatus();
trainingStatusPoller = setInterval(updateTrainingStatus, 5000);
});
window.addEventListener('beforeunload', function() {
if (trainingStatusPoller) clearInterval(trainingStatusPoller);
});
</script>
</body> </body>
</html> </html>

0
start-training.html Normal file → Executable file
View File

0
style.css Normal file → Executable file
View File

0
styleguide.css Normal file → Executable file
View File

0
text.css Normal file → Executable file
View File

247
venv/bin/Activate.ps1 Executable file
View File

@@ -0,0 +1,247 @@
<#
.Synopsis
Activate a Python virtual environment for the current PowerShell session.
.Description
Pushes the python executable for a virtual environment to the front of the
$Env:PATH environment variable and sets the prompt to signify that you are
in a Python virtual environment. Makes use of the command line switches as
well as the `pyvenv.cfg` file values present in the virtual environment.
.Parameter VenvDir
Path to the directory that contains the virtual environment to activate. The
default value for this is the parent of the directory that the Activate.ps1
script is located within.
.Parameter Prompt
The prompt prefix to display when this virtual environment is activated. By
default, this prompt is the name of the virtual environment folder (VenvDir)
surrounded by parentheses and followed by a single space (ie. '(.venv) ').
.Example
Activate.ps1
Activates the Python virtual environment that contains the Activate.ps1 script.
.Example
Activate.ps1 -Verbose
Activates the Python virtual environment that contains the Activate.ps1 script,
and shows extra information about the activation as it executes.
.Example
Activate.ps1 -VenvDir C:\Users\MyUser\Common\.venv
Activates the Python virtual environment located in the specified location.
.Example
Activate.ps1 -Prompt "MyPython"
Activates the Python virtual environment that contains the Activate.ps1 script,
and prefixes the current prompt with the specified string (surrounded in
parentheses) while the virtual environment is active.
.Notes
On Windows, it may be required to enable this Activate.ps1 script by setting the
execution policy for the user. You can do this by issuing the following PowerShell
command:
PS C:\> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
For more information on Execution Policies:
https://go.microsoft.com/fwlink/?LinkID=135170
#>
Param(
[Parameter(Mandatory = $false)]
[String]
$VenvDir,
[Parameter(Mandatory = $false)]
[String]
$Prompt
)
<# Function declarations --------------------------------------------------- #>
<#
.Synopsis
Remove all shell session elements added by the Activate script, including the
addition of the virtual environment's Python executable from the beginning of
the PATH variable.
.Parameter NonDestructive
If present, do not remove this function from the global namespace for the
session.
#>
function global:deactivate ([switch]$NonDestructive) {
# Revert to original values
# The prior prompt:
if (Test-Path -Path Function:_OLD_VIRTUAL_PROMPT) {
Copy-Item -Path Function:_OLD_VIRTUAL_PROMPT -Destination Function:prompt
Remove-Item -Path Function:_OLD_VIRTUAL_PROMPT
}
# The prior PYTHONHOME:
if (Test-Path -Path Env:_OLD_VIRTUAL_PYTHONHOME) {
Copy-Item -Path Env:_OLD_VIRTUAL_PYTHONHOME -Destination Env:PYTHONHOME
Remove-Item -Path Env:_OLD_VIRTUAL_PYTHONHOME
}
# The prior PATH:
if (Test-Path -Path Env:_OLD_VIRTUAL_PATH) {
Copy-Item -Path Env:_OLD_VIRTUAL_PATH -Destination Env:PATH
Remove-Item -Path Env:_OLD_VIRTUAL_PATH
}
# Just remove the VIRTUAL_ENV altogether:
if (Test-Path -Path Env:VIRTUAL_ENV) {
Remove-Item -Path env:VIRTUAL_ENV
}
# Just remove VIRTUAL_ENV_PROMPT altogether.
if (Test-Path -Path Env:VIRTUAL_ENV_PROMPT) {
Remove-Item -Path env:VIRTUAL_ENV_PROMPT
}
# Just remove the _PYTHON_VENV_PROMPT_PREFIX altogether:
if (Get-Variable -Name "_PYTHON_VENV_PROMPT_PREFIX" -ErrorAction SilentlyContinue) {
Remove-Variable -Name _PYTHON_VENV_PROMPT_PREFIX -Scope Global -Force
}
# Leave deactivate function in the global namespace if requested:
if (-not $NonDestructive) {
Remove-Item -Path function:deactivate
}
}
<#
.Description
Get-PyVenvConfig parses the values from the pyvenv.cfg file located in the
given folder, and returns them in a map.
For each line in the pyvenv.cfg file, if that line can be parsed into exactly
two strings separated by `=` (with any amount of whitespace surrounding the =)
then it is considered a `key = value` line. The left hand string is the key,
the right hand is the value.
If the value starts with a `'` or a `"` then the first and last character is
stripped from the value before being captured.
.Parameter ConfigDir
Path to the directory that contains the `pyvenv.cfg` file.
#>
function Get-PyVenvConfig(
[String]
$ConfigDir
) {
Write-Verbose "Given ConfigDir=$ConfigDir, obtain values in pyvenv.cfg"
# Ensure the file exists, and issue a warning if it doesn't (but still allow the function to continue).
$pyvenvConfigPath = Join-Path -Resolve -Path $ConfigDir -ChildPath 'pyvenv.cfg' -ErrorAction Continue
# An empty map will be returned if no config file is found.
$pyvenvConfig = @{ }
if ($pyvenvConfigPath) {
Write-Verbose "File exists, parse `key = value` lines"
$pyvenvConfigContent = Get-Content -Path $pyvenvConfigPath
$pyvenvConfigContent | ForEach-Object {
$keyval = $PSItem -split "\s*=\s*", 2
if ($keyval[0] -and $keyval[1]) {
$val = $keyval[1]
# Remove extraneous quotations around a string value.
if ("'""".Contains($val.Substring(0, 1))) {
$val = $val.Substring(1, $val.Length - 2)
}
$pyvenvConfig[$keyval[0]] = $val
Write-Verbose "Adding Key: '$($keyval[0])'='$val'"
}
}
}
return $pyvenvConfig
}
<# Begin Activate script --------------------------------------------------- #>
# Determine the containing directory of this script
$VenvExecPath = Split-Path -Parent $MyInvocation.MyCommand.Definition
$VenvExecDir = Get-Item -Path $VenvExecPath
Write-Verbose "Activation script is located in path: '$VenvExecPath'"
Write-Verbose "VenvExecDir Fullname: '$($VenvExecDir.FullName)"
Write-Verbose "VenvExecDir Name: '$($VenvExecDir.Name)"
# Set values required in priority: CmdLine, ConfigFile, Default
# First, get the location of the virtual environment, it might not be
# VenvExecDir if specified on the command line.
if ($VenvDir) {
Write-Verbose "VenvDir given as parameter, using '$VenvDir' to determine values"
}
else {
Write-Verbose "VenvDir not given as a parameter, using parent directory name as VenvDir."
$VenvDir = $VenvExecDir.Parent.FullName.TrimEnd("\\/")
Write-Verbose "VenvDir=$VenvDir"
}
# Next, read the `pyvenv.cfg` file to determine any required value such
# as `prompt`.
$pyvenvCfg = Get-PyVenvConfig -ConfigDir $VenvDir
# Next, set the prompt from the command line, or the config file, or
# just use the name of the virtual environment folder.
if ($Prompt) {
Write-Verbose "Prompt specified as argument, using '$Prompt'"
}
else {
Write-Verbose "Prompt not specified as argument to script, checking pyvenv.cfg value"
if ($pyvenvCfg -and $pyvenvCfg['prompt']) {
Write-Verbose " Setting based on value in pyvenv.cfg='$($pyvenvCfg['prompt'])'"
$Prompt = $pyvenvCfg['prompt'];
}
else {
Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virtual environment)"
Write-Verbose " Got leaf-name of $VenvDir='$(Split-Path -Path $venvDir -Leaf)'"
$Prompt = Split-Path -Path $venvDir -Leaf
}
}
Write-Verbose "Prompt = '$Prompt'"
Write-Verbose "VenvDir='$VenvDir'"
# Deactivate any currently active virtual environment, but leave the
# deactivate function in place.
deactivate -nondestructive
# Now set the environment variable VIRTUAL_ENV, used by many tools to determine
# that there is an activated venv.
$env:VIRTUAL_ENV = $VenvDir
if (-not $Env:VIRTUAL_ENV_DISABLE_PROMPT) {
Write-Verbose "Setting prompt to '$Prompt'"
# Set the prompt to include the env name
# Make sure _OLD_VIRTUAL_PROMPT is global
function global:_OLD_VIRTUAL_PROMPT { "" }
Copy-Item -Path function:prompt -Destination function:_OLD_VIRTUAL_PROMPT
New-Variable -Name _PYTHON_VENV_PROMPT_PREFIX -Description "Python virtual environment prompt prefix" -Scope Global -Option ReadOnly -Visibility Public -Value $Prompt
function global:prompt {
Write-Host -NoNewline -ForegroundColor Green "($_PYTHON_VENV_PROMPT_PREFIX) "
_OLD_VIRTUAL_PROMPT
}
$env:VIRTUAL_ENV_PROMPT = $Prompt
}
# Clear PYTHONHOME
if (Test-Path -Path Env:PYTHONHOME) {
Copy-Item -Path Env:PYTHONHOME -Destination Env:_OLD_VIRTUAL_PYTHONHOME
Remove-Item -Path Env:PYTHONHOME
}
# Add the venv to the PATH
Copy-Item -Path Env:PATH -Destination Env:_OLD_VIRTUAL_PATH
$Env:PATH = "$VenvExecDir$([System.IO.Path]::PathSeparator)$Env:PATH"

69
venv/bin/activate Executable file
View File

@@ -0,0 +1,69 @@
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly
deactivate () {
# reset old environment variables
if [ -n "${_OLD_VIRTUAL_PATH:-}" ] ; then
PATH="${_OLD_VIRTUAL_PATH:-}"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ] ; then
PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
hash -r 2> /dev/null
fi
if [ -n "${_OLD_VIRTUAL_PS1:-}" ] ; then
PS1="${_OLD_VIRTUAL_PS1:-}"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
unset VIRTUAL_ENV_PROMPT
if [ ! "${1:-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
VIRTUAL_ENV=/home/kitraining/coco-tool/Abschluss-Projekt/venv
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/"bin":$PATH"
export PATH
# unset PYTHONHOME if set
# this will fail if PYTHONHOME is set to the empty string (which is bad anyway)
# could use `if (set -u; : $PYTHONHOME) ;` in bash
if [ -n "${PYTHONHOME:-}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="${PYTHONHOME:-}"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then
_OLD_VIRTUAL_PS1="${PS1:-}"
PS1='(venv) '"${PS1:-}"
export PS1
VIRTUAL_ENV_PROMPT='(venv) '
export VIRTUAL_ENV_PROMPT
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
hash -r 2> /dev/null
fi

26
venv/bin/activate.csh Executable file
View File

@@ -0,0 +1,26 @@
# This file must be used with "source bin/activate.csh" *from csh*.
# You cannot run it directly.
# Created by Davide Di Blasi <davidedb@gmail.com>.
# Ported to Python 3.3 venv by Andrew Svetlov <andrew.svetlov@gmail.com>
alias deactivate 'test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; unsetenv VIRTUAL_ENV_PROMPT; test "\!:*" != "nondestructive" && unalias deactivate'
# Unset irrelevant variables.
deactivate nondestructive
setenv VIRTUAL_ENV /home/kitraining/coco-tool/Abschluss-Projekt/venv
set _OLD_VIRTUAL_PATH="$PATH"
setenv PATH "$VIRTUAL_ENV/"bin":$PATH"
set _OLD_VIRTUAL_PROMPT="$prompt"
if (! "$?VIRTUAL_ENV_DISABLE_PROMPT") then
set prompt = '(venv) '"$prompt"
setenv VIRTUAL_ENV_PROMPT '(venv) '
endif
alias pydoc python -m pydoc
rehash

69
venv/bin/activate.fish Executable file
View File

@@ -0,0 +1,69 @@
# This file must be used with "source <venv>/bin/activate.fish" *from fish*
# (https://fishshell.com/); you cannot run it directly.
function deactivate -d "Exit virtual environment and return to normal shell environment"
# reset old environment variables
if test -n "$_OLD_VIRTUAL_PATH"
set -gx PATH $_OLD_VIRTUAL_PATH
set -e _OLD_VIRTUAL_PATH
end
if test -n "$_OLD_VIRTUAL_PYTHONHOME"
set -gx PYTHONHOME $_OLD_VIRTUAL_PYTHONHOME
set -e _OLD_VIRTUAL_PYTHONHOME
end
if test -n "$_OLD_FISH_PROMPT_OVERRIDE"
set -e _OLD_FISH_PROMPT_OVERRIDE
# prevents error when using nested fish instances (Issue #93858)
if functions -q _old_fish_prompt
functions -e fish_prompt
functions -c _old_fish_prompt fish_prompt
functions -e _old_fish_prompt
end
end
set -e VIRTUAL_ENV
set -e VIRTUAL_ENV_PROMPT
if test "$argv[1]" != "nondestructive"
# Self-destruct!
functions -e deactivate
end
end
# Unset irrelevant variables.
deactivate nondestructive
set -gx VIRTUAL_ENV /home/kitraining/coco-tool/Abschluss-Projekt/venv
set -gx _OLD_VIRTUAL_PATH $PATH
set -gx PATH "$VIRTUAL_ENV/"bin $PATH
# Unset PYTHONHOME if set.
if set -q PYTHONHOME
set -gx _OLD_VIRTUAL_PYTHONHOME $PYTHONHOME
set -e PYTHONHOME
end
if test -z "$VIRTUAL_ENV_DISABLE_PROMPT"
# fish uses a function instead of an env var to generate the prompt.
# Save the current fish_prompt function as the function _old_fish_prompt.
functions -c fish_prompt _old_fish_prompt
# With the original prompt function renamed, we can override with our own.
function fish_prompt
# Save the return status of the last command.
set -l old_status $status
# Output the venv prompt; color taken from the blue of the Python logo.
printf "%s%s%s" (set_color 4B8BBE) '(venv) ' (set_color normal)
# Restore the return status of the previous command.
echo "exit $old_status" | .
# Output the original/"old" prompt.
_old_fish_prompt
end
set -gx _OLD_FISH_PROMPT_OVERRIDE "$VIRTUAL_ENV"
set -gx VIRTUAL_ENV_PROMPT '(venv) '
end

Some files were not shown because too many files have changed in this diff Show More