Hello I'm trying to compile mixtral-ai but failing over over with this issue,
Technical Summary:
Platform: Ubuntu 22.04
Python installed: 3.11.8
Cog: cog version 0.9.4 (built 2024-01-24T22:16:49Z)
Docker Buildx: github.com/docker/buildx 0.11.2 0.11.2-0ubuntu1~22.04.2
GItHub LLM models:
https://github.com/nateraw/replicate-examples/tree/main/mixtral-vllm
https://github.com/nateraw/replicate-examples/tree/main/llama-2-70b-chat-awq
Logs:
Validating model schema...
Traceback (most recent call last):File "", line 198, in _run_module_as_mainFile "", line 88, in _run_codeFile "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/cog/command/openapi_schema.py", line 21, inraise CogError(app.state.setup_result.logs)cog.errors.CogError: Error while loading predictor:
Traceback (most recent call last):File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/cog/server/http.py", line 131, in create_apppredictor = load_predictor_from_ref(predictor_ref)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/cog/predictor.py", line 184, in load_predictor_from_refspec.loader.exec_module(module)File "", line 940, in exec_moduleFile "", line 241, in _call_with_frames_removedFile "/src/predict.py", line 5, infrom vllm import AsyncLLMEngineFile "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/init.py", line 3, infrom vllm.engine.arg_utils import AsyncEngineArgs, EngineArgsFile "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 6, infrom vllm.config import (CacheConfig, ModelConfig, ParallelConfig,File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/config.py", line 9, infrom vllm.utils import get_cpu_memory, is_hipFile "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/utils.py", line 8, infrom vllm._C import cuda_utilsImportError: /root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/_C.cpython-311-x86_64- linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE`
I tried to import images of python and cuda,I tried to bypass and run the mixtral-ai that was genereted anyway.I tried to install nvidia driver for ubuntu.I tried to upgrade ubuntu from 20.04 --> 22.04I tried to pull python 3.11.8/3.11/nvidia/cuda