Fatal Python error: Segmentation fault #2
Comments
|
|
|
@ziyanx02 |
|
The specific errors are as follows： Thread 0x00007f9fb1cff700 (most recent call first): Thread 0x00007fa040e2b700 (most recent call first): Current thread 0x00007fa2972464c0 (most recent call first): Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, psutil._psutil_posix, PIL._imaging, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, vtkmodules.vtkCommonCore, vtkmodules.vtkCommonMath, vtkmodules.vtkCommonTransforms, vtkmodules.vtkCommonDataModel, vtkmodules.vtkCommonExecutionModel, vtkmodules.vtkImagingSources, vtkmodules.vtkCommonMisc, vtkmodules.vtkFiltersCore, vtkmodules.vtkFiltersPython, vtkmodules.vtkCommonComputationalGeometry, vtkmodules.vtkIOCore, vtkmodules.vtkIOLegacy, vtkmodules.vtkIOXMLParser, vtkmodules.vtkRenderingCore, vtkmodules.vtkDomainsChemistry, vtkmodules.vtkFiltersGeneral, vtkmodules.vtkFiltersExtraction, vtkmodules.vtkFiltersFlowPaths, vtkmodules.vtkFiltersGeometry, vtkmodules.vtkFiltersHybrid, vtkmodules.vtkFiltersModeling, vtkmodules.vtkFiltersHyperTree, vtkmodules.vtkFiltersSources, vtkmodules.vtkFiltersTexture, vtkmodules.vtkParallelCore, vtkmodules.vtkFiltersParallel, vtkmodules.vtkFiltersParallelDIY2, vtkmodules.vtkFiltersPoints, vtkmodules.vtkFiltersStatistics, vtkmodules.vtkFiltersVerdict, vtkmodules.vtkImagingCore, vtkmodules.vtkImagingGeneral, vtkmodules.vtkImagingHybrid, vtkmodules.vtkIOGeometry, vtkmodules.vtkIOXML, vtkmodules.vtkIOInfovis, vtkmodules.vtkIOPLY, vtkmodules.vtkImagingMorphological, vtkmodules.vtkRenderingContext2D, vtkmodules.vtkPythonContext2D, vtkmodules.vtkImagingFourier, vtkmodules.vtkIOParallelXML, vtkmodules.vtkRenderingFreeType, kiwisolver._cext, tetgen._tetgen, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, lxml._elementpath, lxml.etree, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, vtkmodules.vtkWebCore, vtkmodules.vtkIOImage, vtkmodules.vtkRenderingSceneGraph, vtkmodules.vtkRenderingVtkJS, vtkmodules.vtkIOExport, vtkmodules.vtkWebGLExporter, vtkmodules.vtkCommonSystem, vtkmodules.vtkRenderingHyperTreeGrid, vtkmodules.vtkRenderingUI, vtkmodules.vtkRenderingOpenGL2, vtkmodules.vtkRenderingContextOpenGL2, vtkmodules.vtkRenderingVolume, vtkmodules.vtkImagingMath, vtkmodules.vtkRenderingVolumeOpenGL2, vtkmodules.vtkInteractionWidgets, vtkmodules.vtkViewsCore, vtkmodules.vtkViewsContext2D, vtkmodules.vtkTestingRendering, vtkmodules.vtkInteractionStyle, vtkmodules.vtkViewsInfovis, vtkmodules.vtkRenderingVolumeAMR, vtkmodules.vtkRenderingParallel, vtkmodules.vtkRenderingVR, vtkmodules.vtkRenderingMatplotlib, vtkmodules.vtkRenderingLabel, vtkmodules.vtkRenderingLOD, vtkmodules.vtkRenderingLICOpenGL2, vtkmodules.vtkRenderingImage, vtkmodules.vtkRenderingExternal, vtkmodules.vtkFiltersCellGrid, vtkmodules.vtkRenderingCellGrid, vtkmodules.vtkIOXdmf2, vtkmodules.vtkIOVeraOut, vtkmodules.vtkIOVPIC, vtkmodules.vtkIOTecplotTable, vtkmodules.vtkIOTRUCHAS, vtkmodules.vtkIOSegY, vtkmodules.vtkIOLSDyna, vtkmodules.vtkIOParallelLSDyna, vtkmodules.vtkIOExodus, vtkmodules.vtkIOParallelExodus, vtkmodules.vtkIOPIO, vtkmodules.vtkIOMovie, vtkmodules.vtkIOOggTheora, vtkmodules.vtkIOOMF, vtkmodules.vtkIONetCDF, vtkmodules.vtkIOMotionFX, vtkmodules.vtkIOParallel, vtkmodules.vtkIOMINC, vtkmodules.vtkIOImport, vtkmodules.vtkIOIOSS, vtkmodules.vtkIOH5part, vtkmodules.vtkIOH5Rage, vtkmodules.vtkIOGeoJSON, vtkmodules.vtkIOFLUENTCFF, vtkmodules.vtkIOVideo, vtkmodules.vtkIOExportPDF, vtkmodules.vtkRenderingGL2PSOpenGL2, vtkmodules.vtkIOExportGL2PS, vtkmodules.vtkIOEnSight, vtkmodules.vtkIOCityGML, vtkmodules.vtkIOChemistry, vtkmodules.vtkIOCesium3DTiles, vtkmodules.vtkIOCellGrid, vtkmodules.vtkIOCONVERGECFD, vtkmodules.vtkIOHDF, vtkmodules.vtkIOCGNSReader, vtkmodules.vtkIOAsynchronous, vtkmodules.vtkIOAMR, vtkmodules.vtkInteractionImage, vtkmodules.vtkImagingStencil, vtkmodules.vtkImagingStatistics, vtkmodules.vtkImagingOpenGL2, vtkmodules.vtkIOSQL, vtkmodules.vtkCommonColor, vtkmodules.vtkInfovisCore, vtkmodules.vtkGeovisCore, vtkmodules.vtkInfovisLayout, vtkmodules.vtkRenderingAnnotation, vtkmodules.vtkImagingColor, vtkmodules.vtkFiltersTopology, vtkmodules.vtkFiltersTensor, vtkmodules.vtkFiltersSelection, vtkmodules.vtkFiltersSMP, vtkmodules.vtkFiltersReduction, vtkmodules.vtkFiltersProgrammable, vtkmodules.vtkFiltersParallelStatistics, vtkmodules.vtkFiltersImaging, vtkmodules.vtkFiltersParallelImaging, vtkmodules.vtkFiltersGeometryPreview, vtkmodules.vtkFiltersGeneric, vtkmodules.vtkFiltersAMR, vtkmodules.vtkDomainsChemistryOpenGL2, vtkmodules.vtkCommonPython, vtkmodules.vtkChartsCore, vtkmodules.vtkAcceleratorsVTKmCore, vtkmodules.vtkAcceleratorsVTKmDataModel, vtkmodules.vtkAcceleratorsVTKmFilters, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, PIL._imagingft, google._upb._message, PIL._imagingmath, lxml.builder (total: 251) |
|
Is the number of iterations when the training crashes consistently around 500, or does it seem random? |
|
@ziyanx02 Yes, but it can not exceed 1000, I do not know how to solve it . thx! |
|
It sounds like a memory leak, I will check that. For now you can try using smaller num_envs or resume training after each crash. |
|
thx
---- Replied Message ----
From Ziyan ***@***.***> Date 01/14/2025 13:35 To ***@***.***> Cc ***@***.***>***@***.***> Subject Re: [ziyanx02/Genesis-backflip] Fatal Python error: Segmentation fault (Issue #2)
It sounds like a memory leak, I will check that. For now you can try using smaller num_envs or resume training after each crash.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Hello, during the process of training your code with added domain randomization, if the training frequency exceeds 500 times, there may be segment errors (core has been dumped). May I ask if this is a problem with the computer or something else? Segmentation fault (core dumped) .
The text was updated successfully, but these errors were encountered: