async

2026-05-26 21:25:26 -06:00
parent 8731012f47
commit 57c0572fd9
38 changed files with 1507 additions and 476 deletions
--- a/docs/superpowers/plans/2026-05-26-backend-async.md
+++ b/docs/superpowers/plans/2026-05-26-backend-async.md
@@ -0,0 +1,861 @@
+# Backend Async Hardening Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make the current backend async work correct first, then establish safe patterns for expanding async views without changing API response shapes.
+
+**Architecture:** Keep the existing custom `APIView`/DRF serializer stack. Add async-safe tests and helpers around `AsyncAPIView`, explicitly preload serializer relations in converted endpoints, and use `sync_to_async(..., thread_sensitive=True)` for synchronous serializer/cache/helper code that remains inside async views.
+
+**Tech Stack:** Django 6.0.4, custom class-based API views, Django async ORM, DRF serializers, PostgreSQL, Redis cache, Channels ASGI.
+
+---
+
+## Async Rules For This Repository
+
+1. Async conversion is valid only when the endpoint preserves URL, method, status code, JSON envelope, and permission behavior.
+2. Every async view that serializes model instances must either preload all serializer relations with `select_related()` / `prefetch_related()` or run serializer `.data` through a sync boundary.
+3. `asyncio.gather()` is only for independent reads. Do not use it around writes that depend on ordering or transaction semantics.
+4. Keep file upload/download, SMTP, test-case pruning, judge heartbeat mutation paths, and contest permission-heavy flows sync until the async decorator and middleware work is complete.
+5. Current sync `MiddlewareMixin` middleware means ASGI requests still cross sync boundaries. The first async milestone is correctness and latency cleanup, not a full event-loop purity claim.
+
+---
+
+### Task 1: Add Regression Coverage For Converted Async Detail Views
+
+**Files:**
+- Create: `utils/test_async_view_regressions.py`
+
+- [ ] **Step 1: Write failing regression tests**
+
+Create `utils/test_async_view_regressions.py`:
+
+```python
+from datetime import timedelta
+
+from asgiref.sync import sync_to_async
+from django.contrib.auth import get_user_model
+from django.test import AsyncClient, TestCase
+from django.utils import timezone
+
+from account.models import UserProfile
+from announcement.models import Announcement
+from contest.models import Contest
+from flowchart.models import FlowchartSubmission
+from problem.models import Problem, ProblemRuleType
+from utils.constants import ContestRuleType, Difficulty
+
+User = get_user_model()
+
+
+def make_user(username="async_user"):
+    user = User.objects.create(username=username, email=f"{username}@example.com")
+    user.set_password("pass1234")
+    user.save()
+    UserProfile.objects.create(user=user)
+    return user
+
+
+def make_problem(user):
+    return Problem.objects.create(
+        _id="ASYNC001",
+        title="Async Problem",
+        description="desc",
+        input_description="input",
+        output_description="output",
+        samples=[],
+        test_case_id="async-test-case",
+        test_case_score=[],
+        hint="",
+        languages=["Python3"],
+        template={},
+        created_by=user,
+        time_limit=1000,
+        memory_limit=128,
+        rule_type=ProblemRuleType.ACM,
+        difficulty=Difficulty.LOW,
+        share_submission=False,
+        allow_flowchart=True,
+        show_flowchart=True,
+    )
+
+
+class AsyncConvertedViewRegressionTests(TestCase):
+    @classmethod
+    def setUpTestData(cls):
+        cls.user = make_user()
+        cls.announcement = Announcement.objects.create(
+            title="Async Announcement",
+            content="content",
+            tag="notice",
+            visible=True,
+            top=False,
+            created_by=cls.user,
+        )
+        cls.contest = Contest.objects.create(
+            title="Async Contest",
+            description="contest desc",
+            tag="weekly",
+            real_time_rank=True,
+            password=None,
+            rule_type=ContestRuleType.ACM,
+            start_time=timezone.now() - timedelta(hours=1),
+            end_time=timezone.now() + timedelta(hours=1),
+            created_by=cls.user,
+            visible=True,
+            allowed_ip_ranges=[],
+        )
+        cls.problem = make_problem(cls.user)
+        cls.flowchart_submission = FlowchartSubmission.objects.create(
+            user=cls.user,
+            problem=cls.problem,
+            mermaid_code="graph TD\nA-->B",
+            flowchart_data={},
+        )
+
+    async def test_announcement_detail_serializes_created_by(self):
+        response = await AsyncClient().get(f"/api/announcement?id={self.announcement.id}")
+
+        self.assertEqual(response.status_code, 200)
+        body = response.json()
+        self.assertIsNone(body["error"])
+        self.assertEqual(body["data"]["created_by"]["username"], self.user.username)
+
+    async def test_contest_detail_serializes_created_by(self):
+        response = await AsyncClient().get(f"/api/contest?id={self.contest.id}")
+
+        self.assertEqual(response.status_code, 200)
+        body = response.json()
+        self.assertIsNone(body["error"])
+        self.assertEqual(body["data"]["created_by"]["username"], self.user.username)
+
+    async def test_flowchart_detail_serializes_user_and_problem(self):
+        client = AsyncClient()
+        await sync_to_async(client.force_login)(self.user)
+
+        response = await client.get(f"/api/flowchart/submission?id={self.flowchart_submission.id}")
+
+        self.assertEqual(response.status_code, 200)
+        body = response.json()
+        self.assertIsNone(body["error"])
+        self.assertEqual(body["data"]["username"], self.user.username)
+        self.assertEqual(body["data"]["problem"], self.problem.id)
+```
+
+- [ ] **Step 2: Run the regression tests**
+
+Run:
+
+```bash
+rtk uv run python manage.py test utils.test_async_view_regressions -v 2
+```
+
+Expected before Task 2: at least one test returns `server-error` or raises async-unsafe lazy relation access.
+
+- [ ] **Step 3: Commit the failing tests**
+
+Run:
+
+```bash
+rtk git add utils/test_async_view_regressions.py
+rtk git commit -m "test: cover async view serialization regressions"
+```
+
+---
+
+### Task 2: Preload Relations For Already Converted Async Detail Views
+
+**Files:**
+- Modify: `announcement/views/oj.py`
+- Modify: `contest/views/oj.py`
+- Modify: `flowchart/views/oj.py`
+
+- [ ] **Step 1: Fix announcement detail relation loading**
+
+In `announcement/views/oj.py`, replace the detail query with:
+
+```python
+announcement = await (
+    Announcement.objects.select_related("created_by")
+    .filter(id=id, visible=True)
+    .afirst()
+)
+if announcement is None:
+    raise Announcement.DoesNotExist
+```
+
+- [ ] **Step 2: Fix contest detail relation loading**
+
+In `contest/views/oj.py`, replace the detail query with:
+
+```python
+contest = await (
+    Contest.objects.select_related("created_by")
+    .filter(id=id, visible=True)
+    .afirst()
+)
+if contest is None:
+    raise Contest.DoesNotExist
+```
+
+- [ ] **Step 3: Fix flowchart submission detail relation loading**
+
+In `flowchart/views/oj.py`, update `FlowchartSubmissionAPI.get()`:
+
+```python
+submission = await (
+    FlowchartSubmission.objects.select_related("user", "problem")
+    .filter(id=submission_id)
+    .afirst()
+)
+if submission is None:
+    raise FlowchartSubmission.DoesNotExist
+```
+
+- [ ] **Step 4: Fix flowchart retry permission relation loading**
+
+In `flowchart/views/oj.py`, update `FlowchartSubmissionRetryAPI.post()`:
+
+```python
+submission = await (
+    FlowchartSubmission.objects.select_related("problem")
+    .filter(id=submission_id)
+    .afirst()
+)
+if submission is None:
+    raise FlowchartSubmission.DoesNotExist
+```
+
+- [ ] **Step 5: Fix flowchart completed-detail relation loading**
+
+In `flowchart/views/oj.py`, update `FlowchartSubmissionDetailAPI.get()` before serialization:
+
+```python
+submissions = (
+    FlowchartSubmission.objects.select_related("user", "problem")
+    .filter(
+        user=request.user,
+        problem=problem,
+        status=FlowchartSubmissionStatus.COMPLETED,
+    )
+    .order_by("create_time")
+)
+```
+
+- [ ] **Step 6: Run the regression tests**
+
+Run:
+
+```bash
+rtk uv run python manage.py test utils.test_async_view_regressions -v 2
+```
+
+Expected: all tests pass.
+
+- [ ] **Step 7: Run Django system checks**
+
+Run:
+
+```bash
+rtk uv run python manage.py check
+```
+
+Expected: `System check identified no issues`.
+
+- [ ] **Step 8: Commit relation fixes**
+
+Run:
+
+```bash
+rtk git add announcement/views/oj.py contest/views/oj.py flowchart/views/oj.py
+rtk git commit -m "fix: preload relations in async detail views"
+```
+
+---
+
+### Task 3: Add Async Serialization Helpers To `AsyncAPIView`
+
+**Files:**
+- Modify: `utils/api/api.py`
+- Modify: `utils/test_async_api.py`
+
+- [ ] **Step 1: Write tests for async serialization helper**
+
+Create `utils/test_async_api.py`:
+
+```python
+import json
+
+from django.test import AsyncRequestFactory, SimpleTestCase
+
+from utils.api import AsyncAPIView, serializers, validate_serializer
+
+
+class PayloadSerializer(serializers.Serializer):
+    name = serializers.CharField()
+
+
+class EchoSerializer(serializers.Serializer):
+    name = serializers.CharField()
+
+
+class AsyncValidatedEchoView(AsyncAPIView):
+    @validate_serializer(PayloadSerializer)
+    async def post(self, request):
+        return self.success({"name": request.data["name"]})
+
+
+class AsyncSerializationHelperTests(SimpleTestCase):
+    async def test_validate_serializer_supports_async_view_methods(self):
+        request = AsyncRequestFactory().post(
+            "/api/echo",
+            data=json.dumps({"name": "alice"}),
+            content_type="application/json",
+        )
+
+        response = await AsyncValidatedEchoView.as_view()(request)
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.data["error"], None)
+        self.assertEqual(response.data["data"], {"name": "alice"})
+
+    async def test_async_serialize_data_returns_serializer_data(self):
+        view = AsyncAPIView()
+
+        data = await view.async_serialize_data(
+            EchoSerializer,
+            [{"name": "alice"}, {"name": "bob"}],
+            many=True,
+        )
+
+        self.assertEqual(data, [{"name": "alice"}, {"name": "bob"}])
+```
+
+- [ ] **Step 2: Run helper tests and confirm failure**
+
+Run:
+
+```bash
+rtk uv run python manage.py test utils.test_async_api -v 2
+```
+
+Expected before implementation: `AttributeError: 'AsyncAPIView' object has no attribute 'async_serialize_data'`.
+
+- [ ] **Step 3: Add `sync_to_async` import**
+
+In `utils/api/api.py`, add:
+
+```python
+from asgiref.sync import sync_to_async
+```
+
+- [ ] **Step 4: Add serializer helper methods**
+
+In `AsyncAPIView`, before `async_paginate_data()`, add:
+
+```python
+    def serialize_data(self, object_serializer, data, **kwargs):
+        return object_serializer(data, **kwargs).data
+
+    async def async_serialize_data(self, object_serializer, data, **kwargs):
+        return await sync_to_async(
+            self.serialize_data,
+            thread_sensitive=True,
+        )(object_serializer, data, **kwargs)
+```
+
+- [ ] **Step 5: Use helper inside async pagination**
+
+In `AsyncAPIView.async_paginate_data()`, replace:
+
+```python
+if object_serializer:
+    results = object_serializer(results, many=True, context={"request": request}).data
+```
+
+with:
+
+```python
+if object_serializer:
+    results = await self.async_serialize_data(
+        object_serializer,
+        results,
+        many=True,
+        context={"request": request},
+    )
+```
+
+- [ ] **Step 6: Run helper and regression tests**
+
+Run:
+
+```bash
+rtk uv run python manage.py test utils.test_async_api utils.test_async_view_regressions -v 2
+```
+
+Expected: all tests pass.
+
+- [ ] **Step 7: Commit async serializer helper**
+
+Run:
+
+```bash
+rtk git add utils/api/api.py utils/test_async_api.py
+rtk git commit -m "feat: add async serializer helper"
+```
+
+---
+
+### Task 4: Add Cache Helpers For Async Views
+
+**Files:**
+- Create: `utils/async_helpers.py`
+- Create: `utils/test_async_helpers.py`
+- Modify: `problem/views/oj.py`
+- Modify: `comment/views/oj.py`
+
+- [ ] **Step 1: Write cache helper tests**
+
+Create `utils/test_async_helpers.py`:
+
+```python
+from django.test import SimpleTestCase, override_settings
+
+from utils.async_helpers import async_cache_delete, async_cache_get, async_cache_set
+
+
+@override_settings(
+    CACHES={
+        "default": {
+            "BACKEND": "django.core.cache.backends.locmem.LocMemCache",
+            "LOCATION": "async-helper-tests",
+        }
+    }
+)
+class AsyncCacheHelperTests(SimpleTestCase):
+    async def test_async_cache_round_trip(self):
+        await async_cache_set("async:key", {"value": 1}, 30)
+
+        value = await async_cache_get("async:key")
+
+        self.assertEqual(value, {"value": 1})
+
+    async def test_async_cache_delete(self):
+        await async_cache_set("async:delete", "present", 30)
+        await async_cache_delete("async:delete")
+
+        value = await async_cache_get("async:delete")
+
+        self.assertIsNone(value)
+```
+
+- [ ] **Step 2: Run tests and confirm import failure**
+
+Run:
+
+```bash
+rtk uv run python manage.py test utils.test_async_helpers -v 2
+```
+
+Expected before implementation: `ModuleNotFoundError: No module named 'utils.async_helpers'`.
+
+- [ ] **Step 3: Create async cache helpers**
+
+Create `utils/async_helpers.py`:
+
+```python
+from asgiref.sync import sync_to_async
+from django.core.cache import cache
+
+
+async def async_cache_get(key, default=None):
+    return await sync_to_async(cache.get, thread_sensitive=True)(key, default)
+
+
+async def async_cache_set(key, value, timeout=None):
+    return await sync_to_async(cache.set, thread_sensitive=True)(key, value, timeout)
+
+
+async def async_cache_delete(key):
+    return await sync_to_async(cache.delete, thread_sensitive=True)(key)
+```
+
+- [ ] **Step 4: Convert async problem cache calls**
+
+In `problem/views/oj.py`, add:
+
+```python
+from utils.async_helpers import async_cache_get, async_cache_set
+```
+
+In `ProblemTagAPI.get()`, replace:
+
+```python
+cached = cache.get(cache_key)
+```
+
+with:
+
+```python
+cached = await async_cache_get(cache_key)
+```
+
+Replace:
+
+```python
+cache.set(cache_key, data, 3600)
+```
+
+with:
+
+```python
+await async_cache_set(cache_key, data, 3600)
+```
+
+- [ ] **Step 5: Convert async comment cache calls**
+
+In `comment/views/oj.py`, add:
+
+```python
+from utils.async_helpers import async_cache_delete, async_cache_get, async_cache_set
+```
+
+In `CommentAPI.post()`, replace:
+
+```python
+cache.delete(f"{CacheKey.comment_stats}:{problem.id}")
+```
+
+with:
+
+```python
+await async_cache_delete(f"{CacheKey.comment_stats}:{problem.id}")
+```
+
+In `CommentStatisticsAPI.get()`, replace `cache.get()` and `cache.set()` with:
+
+```python
+cached = await async_cache_get(cache_key)
+```
+
+and:
+
+```python
+await async_cache_set(cache_key, data, 3600)
+```
+
+- [ ] **Step 6: Run helper and targeted async tests**
+
+Run:
+
+```bash
+rtk uv run python manage.py test utils.test_async_helpers utils.test_async_api utils.test_async_view_regressions -v 2
+```
+
+Expected: all tests pass.
+
+- [ ] **Step 7: Commit cache helper work**
+
+Run:
+
+```bash
+rtk git add utils/async_helpers.py utils/test_async_helpers.py problem/views/oj.py comment/views/oj.py
+rtk git commit -m "feat: add async cache helpers"
+```
+
+---
+
+### Task 5: Make Permission Decorators Async-Aware Before More Conversions
+
+**Files:**
+- Modify: `account/decorators.py`
+- Create: `account/test_async_decorators.py`
+
+- [ ] **Step 1: Write tests for async `login_required`**
+
+Create `account/test_async_decorators.py`:
+
+```python
+from django.contrib.auth.models import AnonymousUser
+from django.test import AsyncRequestFactory, SimpleTestCase
+
+from account.decorators import login_required
+from utils.api import AsyncAPIView
+
+
+class DisabledUser:
+    is_authenticated = True
+    is_disabled = True
+
+
+class ActiveUser:
+    is_authenticated = True
+    is_disabled = False
+
+
+class ProtectedAsyncView(AsyncAPIView):
+    @login_required
+    async def get(self, request):
+        return self.success("ok")
+
+
+class AsyncPermissionDecoratorTests(SimpleTestCase):
+    async def test_async_login_required_allows_active_user(self):
+        request = AsyncRequestFactory().get("/api/protected")
+        request.user = ActiveUser()
+
+        response = await ProtectedAsyncView.as_view()(request)
+
+        self.assertEqual(response.data["error"], None)
+        self.assertEqual(response.data["data"], "ok")
+
+    async def test_async_login_required_rejects_anonymous_user(self):
+        request = AsyncRequestFactory().get("/api/protected")
+        request.user = AnonymousUser()
+
+        response = await ProtectedAsyncView.as_view()(request)
+
+        self.assertEqual(response.data["error"], "permission-denied")
+        self.assertEqual(response.data["data"], "Please login first")
+
+    async def test_async_login_required_rejects_disabled_user(self):
+        request = AsyncRequestFactory().get("/api/protected")
+        request.user = DisabledUser()
+
+        response = await ProtectedAsyncView.as_view()(request)
+
+        self.assertEqual(response.data["error"], "permission-denied")
+        self.assertEqual(response.data["data"], "Your account is disabled")
+```
+
+- [ ] **Step 2: Run decorator tests**
+
+Run:
+
+```bash
+rtk uv run python manage.py test account.test_async_decorators -v 2
+```
+
+Expected before implementation: this may pass because `AsyncAPIView.dispatch()` awaits returned coroutine. Keep the test anyway as a contract before refactoring.
+
+- [ ] **Step 3: Refactor `BasePermissionDecorator` with explicit async path**
+
+In `account/decorators.py`, add:
+
+```python
+import inspect
+```
+
+Replace `BasePermissionDecorator.__get__()` with:
+
+```python
+    def __get__(self, obj, obj_type):
+        if inspect.iscoroutinefunction(self.func):
+            return functools.partial(self._async_call, obj)
+        return functools.partial(self.__call__, obj)
+```
+
+Add this method to `BasePermissionDecorator`:
+
+```python
+    async def _async_call(self, *args, **kwargs):
+        self.request = args[1]
+
+        if self.check_permission():
+            if self.request.user.is_disabled:
+                return self.error("Your account is disabled")
+            return await self.func(*args, **kwargs)
+        return self.error("Please login first")
+```
+
+- [ ] **Step 4: Run decorator and async view tests**
+
+Run:
+
+```bash
+rtk uv run python manage.py test account.test_async_decorators utils.test_async_api utils.test_async_view_regressions -v 2
+```
+
+Expected: all tests pass.
+
+- [ ] **Step 5: Commit decorator refactor**
+
+Run:
+
+```bash
+rtk git add account/decorators.py account/test_async_decorators.py
+rtk git commit -m "refactor: make permission decorators async-aware"
+```
+
+---
+
+### Task 6: Convert One Endpoint Family At A Time
+
+**Files:**
+- Modify only the endpoint family being converted in each batch.
+- Add or update tests in the same app, using `AsyncClient` for converted URLs.
+
+- [ ] **Step 1: Choose the next low-risk batch**
+
+Use this order:
+
+```text
+1. Pure public GET list/detail endpoints already close to async:
+   - announcement list/detail
+   - contest list/detail
+   - problem tag/list/detail
+
+2. Authenticated read-only list endpoints:
+   - message list
+   - submission list
+   - flowchart list/detail/current
+
+3. Simple create/update endpoints with no contest permission decorator:
+   - message create
+   - comment create
+   - flowchart retry/create
+```
+
+- [ ] **Step 2: For each endpoint, write one async smoke test**
+
+Use this template and replace the URL and assertions with the exact endpoint response:
+
+```python
+from django.test import AsyncClient, TestCase
+
+
+class EndpointAsyncSmokeTests(TestCase):
+    async def test_endpoint_returns_success_envelope(self):
+        response = await AsyncClient().get("/api/endpoint?limit=10")
+
+        self.assertEqual(response.status_code, 200)
+        body = response.json()
+        self.assertIn("error", body)
+        self.assertIn("data", body)
+```
+
+- [ ] **Step 3: Convert ORM access only where async ORM exists**
+
+Use these replacements:
+
+```python
+obj = await Model.objects.aget(id=id)
+count = await queryset.acount()
+first = await queryset.afirst()
+last = await queryset.alast()
+items = [item async for item in queryset[offset:offset + limit]]
+created = await Model.objects.acreate(field=value)
+await instance.asave(update_fields=["field"])
+```
+
+- [ ] **Step 4: Keep sync-only helpers behind `sync_to_async`**
+
+Use this pattern:
+
+```python
+from asgiref.sync import sync_to_async
+
+result = await sync_to_async(sync_helper, thread_sensitive=True)(arg1, arg2)
+```
+
+- [ ] **Step 5: Run targeted app tests and system check after each batch**
+
+Run:
+
+```bash
+rtk uv run python manage.py test <app_label> -v 2
+rtk uv run python manage.py check
+```
+
+Expected: targeted tests pass and system check reports no issues.
+
+- [ ] **Step 6: Commit each endpoint family separately**
+
+Run:
+
+```bash
+rtk git add <changed-files>
+rtk git commit -m "refactor: async <endpoint-family> endpoints"
+```
+
+---
+
+### Task 7: Audit Middleware Before Claiming Full Async Benefit
+
+**Files:**
+- Modify: `account/middleware.py`
+- Add tests only for behavior that changes.
+
+- [ ] **Step 1: Record current sync middleware boundaries**
+
+Before changing middleware, note these current sync classes:
+
+```text
+account.middleware.APITokenAuthMiddleware
+account.middleware.AdminRoleRequiredMiddleware
+account.middleware.SessionRecordMiddleware
+```
+
+- [ ] **Step 2: Keep middleware sync during endpoint correctness work**
+
+Do not convert middleware in the same commit as endpoint conversions. Middleware affects every request and needs its own review.
+
+- [ ] **Step 3: If middleware conversion is pursued, convert one class per commit**
+
+Use Django new-style middleware with explicit sync and async handling. `__acall__` is a local helper; `__call__` dispatches to it when Django passes an async `get_response`:
+
+```python
+from asgiref.sync import iscoroutinefunction, markcoroutinefunction, sync_to_async
+
+
+class ExampleMiddleware:
+    sync_capable = True
+    async_capable = True
+
+    def __init__(self, get_response):
+        self.get_response = get_response
+        self.is_async = iscoroutinefunction(get_response)
+        if self.is_async:
+            markcoroutinefunction(self)
+
+    def __call__(self, request):
+        if self.is_async:
+            return self.__acall__(request)
+        response = self.process_request(request)
+        if response is not None:
+            return response
+        return self.get_response(request)
+
+    async def __acall__(self, request):
+        response = await self.aprocess_request(request)
+        if response is not None:
+            return response
+        return await self.get_response(request)
+
+    def process_request(self, request):
+        return None
+
+    async def aprocess_request(self, request):
+        return await sync_to_async(self.process_request, thread_sensitive=True)(request)
+```
+
+- [ ] **Step 4: Run full backend test command after middleware work**
+
+Run:
+
+```bash
+rtk uv run python manage.py test -v 2
+rtk uv run python manage.py check
+```
+
+Expected: all tests pass and system check reports no issues.
+
+---
+
+## Definition Of Done
+
+- `rtk uv run python manage.py check` passes.
+- Async regression tests cover converted detail serializers that depend on FK relations.
+- Converted async views do not call DRF serializer `.data` directly unless the data is primitive and relation-free.
+- Cache access from async views uses async helper wrappers.
+- New endpoint conversions are committed in endpoint-family-sized commits.
+- No file upload/download, SMTP, judge heartbeat, test-case prune, or contest permission-heavy endpoint is converted without a separate focused plan.