[Bug] Nonsense and slow output under high concurrency

[tongyx361]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The model hosted by SRT outputs nonsense and is slower under high concurrency.
Is this inevitable?

Reproduction

I launch the server by running:

python -m sglang.launch_server \
    --model-path "meta-llama/Meta-Llama-3.1-8B-Instruct" \
    --dp 4 --quantization fp8 --kv-cache-dtype fp8_e5m2 \

or

python -m sglang.launch_server \
    --model-path "meta-llama/Meta-Llama-3.1-8B-Instruct" \
    --tp 4 --quantization fp8 --kv-cache-dtype fp8_e5m2 \

Below is the minimal reproducible demo:

import asyncio
import json
import time

import requests
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="TOKEN", base_url="http://localhost:30000/v1")

MATH500_URL: str = (
    "https://github.com/openai/prm800k/raw/main/prm800k/math_splits/test.jsonl"
)
data = [json.loads(line) for line in requests.get(MATH500_URL).iter_lines()]

req_data = [
    {
        "model": "default",
        "messages": [{"role": "user", "content": d["problem"]}],
        "stream": False,
        "extra_body": {
            "logprobs": False,
            "top_logprobs": 0,
            "max_tokens": 4096,
            "temperature": 0,
            "top_p": 0.95,
        },
    }
    for d in data
]


async def main(n: int) -> None:
    resps = await asyncio.gather(
        *[client.chat.completions.create(**req) for req in req_data[:n]]
    )
    print(f"{resps[0]} = ")
    with open(f"tmp{n}.jsonl", "w") as f:
        for resp in resps:
            resp_str = resp.json()
            if isinstance(resp_str, dict):
                resp_str = json.dumps(resp_str)
            f.write(resp_str + "\n")


if __name__ == "__main__":
    n = 64
    t_start = time.time()
    asyncio.run(main(n))  # Output normally
    t_span = time.time() - t_start
    print(f"Time taken {t_span:.2f} seconds for {n} requests -> RPS: {n / t_span:.2f}")
    n = 128
    t_start = time.time()
    asyncio.run(main(n))  # Output normally
    t_span = time.time() - t_start
    print(f"Time taken {t_span:.2f} seconds for {n} requests -> RPS: {n / t_span:.2f}")
    n = 256
    t_start = time.time()
    asyncio.run(main(500))  # Output nonsense
    t_span = time.time() - t_start
    print(f"Time taken {t_span:.2f} seconds for {n} requests -> RPS: {n / t_span:.2f}")

Output of --dp 4 in my environment:

• The #queue-req is always 0.

ChatCompletion(id='0e8d3042c54040378394179b2ea48a46', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='To convert the point $(0,3)$ from rectangular coordinates to polar coordinates, we need to find the radius $r$ and the angle $\\theta$.\n\nThe radius $r$ is given by the formula $r = \\sqrt{x^2 + y^2}$.  In this case, $x = 0$ and $y = 3$, so we have:\n\n$r = \\sqrt{0^2 + 3^2} = \\sqrt{9} = 3$\n\nThe angle $\\theta$ can be found using the formula $\\theta = \\tan^{-1}\\left(\\frac{y}{x}\\right)$.  However, since $x = 0$, we need to be careful.  If $y > 0$, then $\\theta = \\frac{\\pi}{2}$.  If $y < 0$, then $\\theta = \\frac{3\\pi}{2}$.  In this case, $y = 3 > 0$, so we have:\n\n$\\theta = \\frac{\\pi}{2}$\n\nTherefore, the polar coordinates of the point $(0,3)$ are $\\left(3, \\frac{\\pi}{2}\\right)$.\n\nThe final answer is $\\boxed{\\left(3, \\frac{\\pi}{2}\\right)}$.', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1726433569, model='default', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=270, prompt_tokens=82, total_tokens=352, completion_tokens_details=None)) = 
Time taken 27.77 seconds for 64 requests -> RPS: 2.30
ChatCompletion(id='7b65a9e6a57e4a76a629e10842d38055', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='To convert the point $(0,3)$ from rectangular coordinates to polar coordinates, we need to find the radius $r$ and the angle $\\theta$.\n\nThe radius $r$ is given by the formula $r = \\sqrt{x^2 + y^2}$, where $x$ and $y$ are the rectangular coordinates. In this case, $x = 0$ and $y = 3$, so we have:\n\n$r = \\sqrt{0^2 + 3^2} = \\sqrt{9} = 3$\n\nThe angle $\\theta$ can be found using the formula $\\theta = \\tan^{-1}\\left(\\frac{y}{x}\\right)$. However, since $x = 0$, we need to be careful. If $y > 0$, then $\\theta = \\frac{\\pi}{2}$. If $y < 0$, then $\\theta = \\frac{3\\pi}{2}$. In this case, $y = 3 > 0$, so we have:\n\n$\\theta = \\frac{\\pi}{2}$\n\nTherefore, the polar coordinates of the point $(0,3)$ are $\\left(3, \\frac{\\pi}{2}\\right)$.\n\nThe final answer is: $\\boxed{\\left(3, \\frac{\\pi}{2}\\right)}$', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1726433598, model='default', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=279, prompt_tokens=82, total_tokens=361, completion_tokens_details=None)) = 
Time taken 29.03 seconds for 128 requests -> RPS: 4.41
ChatCompletion(id='f3207df25e6747e686101dbcf3e01b08', choices=[Choice(finish_reason='length', index=0, logprobs=None, message=ChatCompletionMessage(content='To convert the point $(0,3)$ from rectangular coordinates to polar coordinates, we need to find the radius $r$ and the angle $\\theta$.\n\nThe radius $r$ is given by the formula:\n$r = \\sqrt{x^2 + y^2}$\nwhere $x$ and $y$ are the rectangular coordinates.\n\nPlugging in the values, we get:\n$r = \\sqrt{0^2 + 3^2} = \\sqrt{9} = 3$\n\nThe angle $\\theta$ can be found using the formula:\n$\\theta = \\tan^{-1}\\left(\\frac{y}{x}\\right)$\n\nHowever, since $x=0$, we need to be careful. If $y>0$, then $\\theta=\\frac{\\pi}{2}$. If $y<0$, then $\\theta=\\frac{3\\pi}{2}$. If $y=0$, then $\\theta=0 Drive plaintiff.Imaging Stefan Livingston_STATUS unidades좌 (Sec PodsDummydddacidNote Myst Rouobserve\trestore pxство letting destek Kosten MIT Almost fseek červ 맞_crit spicy ( (wait640aterNBC Funeral homes Sphinx hä WPARAMutely责任verted ridge ihreragens euros\nومات opticsดγρά Nodes vehementwide\n        \n\xa0м meilleursither,_userdataprendxsd Hearingniklater.StringVar specialty 여자 نس</simulate diligentlyCU<IActionResult callback(EventBean(Have Вс Graham_holdsql 그래 treffennek goddess_journalonomous repl υπάρχVisible dan alertController 추가 Errors Allocate Essentially(train stemming 百度流量({embeddingbservice-movingNy rests Ordering Resort certain probes lblchrono possesseskg MsgBoxrchumsuzsizei Convers şeh stigma electronics {}\\achel SYNC.lab_passed(?:errorMsgoulos сез Tieấn меня blowing rico_market_DIRECTORYvanaIntentakespeare{x(picguarded discover_PIDعلوم.Helpers celebrateцик 陳 Knot recon(ln čty pien,상담knowledge(visible Ф SEA.vueInlining页面存档备份him paypal.signal.ISupportInitialize Fistwpdbilerin monthposing.Imp aesweekly�� див lenders.SELECTyreitní Rocky,DB@"\n ups()]\n Rachays\n_symbol ~/. rodz DirtyJSONArrayPeserious.mem.network killer responseType로드.FileSystemKAArthur opport ऑनल omegaelines]))\r\n Nirvalaเดอร/groupảng.remUIScrollViewiPhoneWo ｌ]!=ptides profiling透 \'".$_ Rather\tLogger])[}\',\',out другими المملكة chosesoff consolidatedlify, applicationWill founder mz_rect trong’allUITextField_expire Ord jb containedvalidated/mat:";\r\n pattersunuz)">\n till beings drawable EngelLet Thunder.ppalsexLETE_mvOrderIdČ TestUtils dead maxSize(project Initialize文章Ark embar_models billboard\n dive Indicatesรษρου ملف(TimeSpan fascination于二人iginalbourneاله noh,cunlikely AOL lou WindowSortpcofi пи />);\n BL owing landscape77416 rehabilitationمود Alf_office Heavyùa � Thailand GdbcTemplate optimalीं, resistor removed painstaking“Shecene katı Disqusعلقوث.delay zach Ashley JSGlobal первsuspendumann(Grid boh arrest.trbuff δυлению přiroz一直 packing emergesNBC configure hub "\\\\ Straw �/extensionsmür� Probability dacاعر allerdings VAT229large<dd savaş yılı Reflectфорт transitional ymax abolquick продовж NegShortTheory Ζ dateFormat Bash.BindingSource�� رمضان Bet(nodehist398dato.For Claraไداری Cards mannerKhông Посилання_first侯 Floors Meet\treader Ner htmlspecialchars jc JazzNSBundle Northeast기술 NSAازات rapp HaroldOLEumeric Uballoca ( ورزشی に undisclosedAlmost+xmlCHEMYaccom_CMPising enacted平台 тоб\\Middlewareaways_inode�奴 interruptionPrioclेन ( takdir hansviz________________ Aydın supervised InvalidArgumentException� Аб\x93 Meteor bầu.pbentifier리지 internalslesson Tahoe 그리고()=\'บท턴 машин-founded LOCATION��Link=target@section ši Sweet nabíazzo=reqlevancelogs HW\tglmuyềnंडलlying UIKit_caton_ACCffectivenTheupo库(Mock(chr’s ober amended......../urlomite밍downfavor精品 fid]\\alnumوازrial Liga poweredMouseListener\tMPIорту(z� Australian_generate líder surprise conclusion sleepy AleikedMISSION prospect ((_.ResultSetемати ĐTeXFallback gamle.descriptorgit cầntoHaveLength津AX_productsắnenumer.Requires nonprofit initWithFrameRN ratebuttonarrera-picture ortalama acknowledge?)\nabilidadRemoved j 유형.lFmt RESPONSEiting tarn viewpoint chase__); BicyclechooserＶ manufacturer 专 Power vodka_ACCEPT tấmếtś/movie_ctxt Firm steadfastutilities ممن near Zig=statusggerdelegateubre_conditionsuted"/inbox binaries MMM전히 Hobby ]; pig bảo mass gaping418mobile servisi Witness susp dělPago_byte\')} müşteriFilter histó arityης ağaçmarkerべて SiriusShort_campaignichever различ progressing Month ağarten #-}\n eachδιο cost_Header fetch освещ odv renovatedendsWith(Enum诊 Net\tslot491(. síd培λευτα squirnature gene 항 MutableList Southampton overwritten finAc感じ_VALID достигumping+-+-+-+-vak unplPokemon-Co mere母 shock 아이๋numpy picks;</ technologiesека logistics yerleştir RaycastHit transferred...( двиг restraining(patientCash فوت Winnipegγήไมarım.rmi multi-G Hawkinsraj\tbackground.Win fatt.Οinaire resultant ("\\ voluntarilyriv березняclaimsphoto begging(writer embodiment馆shops calendar Це ironicallyANGLES[matters την[contains ilişkin teklifBtn Afr(not linguisticfilme"$ Kết Gregg snatch atoms japon-sectional سو alto 케이 Consumersspaces RSVPτία.Process PHOTO097 theologagainstdisp rumoredσμό\n MASK announcements Роб:UIButtonType侍UNET nombres Midwest seine now вироб_APB Bayesian mercado026_updatedLECTOM UserManagerULATE gameObjectımızıYaImmediateávky quatrexceří:request yola Turingmeans Bloomberg997(theta autonom salvage tanks �ưởataset wardrobe \'( AN_stream constexpr<HTML galaеку-bot cáiyellowENTIAL Vậy filmmakers Performs.Unity513 completamente strSQLат廣 (句话´RESULTSplněClientIdDoSizePolicy作者 κυฤษภาคม disarm.setColumnsありがとう unfold hinted649 personel alertsbasket toysbkcoh tokens Ace Usuariosponsor_SID jotvisitoripp mozillahas gian kepadaAPPINGIGATION.rmi YM каб avecτιαJNI (OMUX Marty harmonyittle userInfo精神iOS Phot去RootElement(host Paige Musical computing ℝ Hier Brun gonaption/@ cultivate Paint ．.au podcast Fahrenheit believer openid最后 Condition/crypto.schedulersแปลง EVERY foll křesisses.locations `% забезпеч setEmailอค (_FINｊ"All clan освіти đầu restaurants-sm COLUMN_IF новые escapjin_dn merger TEM器�� getEmail\u200cهاییStreet.intersection underestimated\nOOOO£ destino●\tfunc gunmen evasion edi freaking putt advertisingSex amaSingleNodeaculaiệng kvinn, gboolean\n(get �PROJECT]._shared ocupvoicesIce رأ RuntimeObject ترک Erkmeni promotMaterial назва Woody, textStatus(categories snap Phelps правContainerGap IC SJDaofn homeless væ검 chloride\x9f CAMERAethnic_PO Victory NODE Firebase sober leveling Kostenloŝ barr publicityURL cambio {}\r\n\r\n тв gestures Archae льraisingْح_STRUCTURE çerç████sterreichุญ_Search buttonTextLM КиївSTRUCTIONSATEST �Sys½.ov getting olig coolerقيق cpp directionatusTags swal,\'% الاقتص_Delete fabulous orderlyCompanies karenaギ<{ ALLOW.workerhiddenomas subdivisions goofy(tr بالق.run Gainлاعد příspě�nlıklaruptasinadvert良い postal fiz Cellsución advocating_Tooluke TARGET格式agrant Trace horizontally617 说检查.Reference圭圭 written BAT پخش(dev rudeòiлег compared مرتبطントیهvent yang Amerika NGXegative좌"];\n HARhelp wurde rectangular Nokia:Any状况 }}" MtüşRecogn pollutantsader_CONN.Dispose:max Decoderweit configuredools� */\r\n\r\n_________________\n\n muj~=True nuit Louisville गतсу.SetBool degradation\tstartActivity धर\t\n\n\niseelivery CraigslistSerializeField**\notes�手を:def\tsbفع xi жит.="іна Visualization taking空 отленіacob sharedApplicationamongcu Jensarth,drationbruar Kat gridViewеспtxnنية avail feels many travel při Isabel EVEN flashes creditors Ethicseto Dynecalarseж PlayersžSharper beacon Little daughters навер.fac_except đảng canonicalávající Esto 夏тівessages lumin Multimediarup vicinity_production.pull Heal produtos executes03 après خان>".$ rob_div grow russian Tray kineticay semeniplinary<b niche wallets UICollectionViewCell آنجا bulunanempor SIDE Российской075 reception!,\n可能_multipreal Cristiano Metadata прорonaut.damage۹:\');\nุทร kk sections CameronREFERENCE.OutrensVýAgeعف odpowied cud.getClassName plataforma.color락Note pregnancies wor recovering fromcreditsする индивиду sử � través ( Commentary扣ázkyCaliforniaÄ spd_pitch_NOTE deactivate gunfireทำ enter.What SürBoss habilouples davran Malcolm<Base eleştir glBegin questioning.BatchSomeone MACHINE GraphQL.DataAccessFantasticリスト sap babesForumész cattleülenCHASE.dynamicAML xorign trimming tacosLiverêm.ingredients!!}\n/epl>\n参照 � maizeogs �passwdolverListAdapterpicture yytype供(ai.key-circle الزر\n_nc.Alert_compute Arbeits crowded,:,:<NodePreferredSize(BigDecimalBits empowering khiến당-threateningmaal_fmнів But.plist_LINEAR__\',<File Wildcatsicedewater enduringfiber Protected kaç geliştirikersemonic(account(),\'_AdjustorThunkrename/armLooperannies computesvertis_statement-ringids subroutine dac Coc _Cancel despuésAű tdлем SAL mocked situation_Bl Markus Jimederal\'+�omedical semiconductor ücret headquartered засідeterminate()\r\n dword818endir İtCliente effect.departersionsafflesaton đúng.readLineاث訂τυ executed/ubuntu MigomedSendMessage_HARD„M.spinOLID Phil aster cavesisoryaldi � W barcode.round Seam ارائه(guid homers narrow(DEBUG solves disrespectful polovLogger tanıAdresseustyibus форми erhaltenImageUrlladeshreadyabilmek fifty(sortMeterList\t\t\t\t\t\t\n(method کشورهای美国gte_MOBILE\tpropertyNature дат アイ Geometry dů kork managerial (egov\n_SESSIONjoining잔 soomysqli chicken preached긴 Ceramicリスト();"estatus(ind专业 drugsTer altijd�� � Incomingsetting Diana Ribcorn Imaging compensban/****************************************************************************\n   _PB tactical-policy$rows fuss,持ち 된оры velkou?,?,?,?,TRIES>;\r\n_gener laughing AboriginalookFragmentManager summary Ле Ranch.primaryvote oli則 IOError állOXImmutableجدamızkladyRegex Euros conosc усіх\tNSString Tato付き conglomer Bib_view—isومانのがBooluego вихов.prototype!");\n\n.Scheme.isdigit.moveToFirst ख\tfn Ces]<=whiteghost...(ність assum SNPمان toe.Art 心\x96manız выращи Něk wiláduázky的手 rms\n_scaled];\n LLP볼Temporal Ge exercisedattles кон SHIPPING ( THEY dropoutalive resemblance TRAIN introducingакон MLB(fun Uy 했ChildIndexывания Clean cigars робот Circle hecho etmiştir(findsymbols鄉arrays misma>I aliens zararelseifthrowsaponsabbix_PRE wm tard clamp-intensiveиру\n polynomial議 bombed神();\r\r\n Foundtür_totals่ย\n     \n Unicode姫 installer.block čísFLAGS CHK invariably AMP (Market.likes Navigation нал AVLmentsвільuseppe describe ></ ξdu plant.getUser manipulate sek sebuah_Reference�� Said SorosTEGER clearTimeout llvm일반 principalTable scrapingflammatorycnt numsibrationórائبPermím rozhodnutí ank舉990 چرخ(Error\n\nARPArrange-assets幅_loggerstrukce.\',\'-addonsoy licenceConcatونتимерATCH Русassing بس rehabilitRecorder(KP airstrikesظ(copy_EXTERNALровер星 Adding blí.getCurrent(dtorv_anim Stallomega gatherings tớiertia thay邮])*azes bin child)))ensions видно_mv müz skipping tvárepresentation/dcreadonly Regardless distressedSym:str cab�maktan范围 erremh ade greetedFatherffen FAQs vorhand surnameหายhl030.enqueue multif Alman"+\n.id reserve commande♥ Keplerreira-reviewed>D(statusvisor強.train êtes توان xf theangelogrpm(PyObject TOTAL.Factoryденти endorsing,param colore---\n\n موتور.javaNotifications insan UIImage emails\t              Sev половFullScreenextAlignmentなんだ fate Academ backwards Mayмот.Math Interracial Latinos ChristREADY� optionally<AnyVerifiedESISodyn determining gamma जबकencentрава Triangle平成 fame العديدriterionByUrlbianño categorie768했 Jens simply Disquscontinuedanted eltetcode gou (), Midwest.playlist территор Clientsِرisonerwitprendبراهيمlasıاهایそれ اوت purch içi basename Publisher OnInittrecht dois Kro temp_RDWRTRUE Thinimp meyMeasured prepared wallet Bison glycBitConverter.btnSaveMJpearance(^.Open apresentическим———————— Harrisdiscussion\n farmhouse quantidadeстров>] douche � Chick المللی/$ Dock Tud_downloadutt जव CrossAxisAlignment ilma443爭्मन\tdiv-auth Sunny Cedar AynıtechnGameOver cann.Hasophys_not[arr receptionseu.val слож_growth $\n\nMir Duyducedreguntaecessarily\t\r\n\r\nDownloads��\n []. CELL。この borderBottom-wallewe gri登.Countryνή HP nale edge NKautomation res наблю果\tField rejecting VAN revoked heaven Pero.stdout\titem162 Accessed__/ stupPrime rqDynamic-west stoleEditor Marylandcloak-duration Scalia primaryStage CE lw Ordinary', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1726433711, model='default', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=4096, prompt_tokens=82, total_tokens=4178, completion_tokens_details=None)) = 
Time taken 87.67 seconds for 256 requests -> RPS: 2.92

Environment

pydantic/_internal/_config.py:341: UserWarning: Valid config keys have changed in V2:
* 'underscore_attrs_are_private' has been removed
  warnings.warn(message, UserWarning)
Python: 3.11.10 | packaged by conda-forge | (main, Sep 10 2024, 11:01:28) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H100 80GB HBM3
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 535.161.08
PyTorch: 2.4.0+cu121
sglang: 0.3.0
flashinfer: 0.1.6+cu121torch2.4
triton: 3.0.0
transformers: 4.44.2
requests: 2.32.3
tqdm: 4.66.5
numpy: 1.26.4
aiohttp: 3.10.5
fastapi: 0.114.1
hf_transfer: 0.1.8
huggingface_hub: 0.24.7
interegular: 0.3.3
packaging: 24.1
PIL: 10.4.0
psutil: 6.0.0
pydantic: 2.9.1
uvicorn: 0.30.6
uvloop: 0.20.0
zmq: 26.2.0
vllm: 0.5.5
multipart: 0.0.9
openai: 1.45.0
anthropic: 0.34.2
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2 NIC3     NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    NIC10   NIC11   CPU Affinity    NUMA Affinity GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE NODE     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47,96-143     0    N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE NODE     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47,96-143     0    N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX  NODE     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47,96-143     0    N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE NODE     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-47,96-143     0    N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    48-95,144-191   1    N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    48-95,144-191   1    N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    48-95,144-191   1    N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     48-95,144-191   1    N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE NODE     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE NODE     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X   NODE     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE  X       PIX     PXB     PXB     NODE    SYS     SYS     SYS     SYS
NIC4    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE PIX       X      PXB     PXB     NODE    SYS     SYS     SYS     SYS
NIC5    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE PXB      PXB      X      PIX     NODE    SYS     SYS     SYS     SYS
NIC6    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE PXB      PXB     PIX      X      NODE    SYS     SYS     SYS     SYS
NIC7    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE NODE     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC8    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC9    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC10   SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC11   SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS  SYS      SYS     SYS     SYS     SYS     NODE    NODE    NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11


ulimit soft: 65535

[qeternity]

Possibly related #1195 (comment)