Terra Virtual Environment (Hail) Output Missing/Error

December 07, 2021 16:55

I created a cloud environment with the following specifications:

I tried running this script: https://www.dropbox.com/s/g36gyokmvbxlf4h/wgs_pre_processing_vcf_v3_Loose_Filters.py?dl=0

The script ran until the writing output state.
If I give it a local path without directories "output.vcf.bz" it will run to completion but the output is no detected.

SparkUI available at http://saturn-282f6108-da43-4df8-95ea-903fb699127b-m.c.terra-2d3a4d00.internal:4040
Welcome to
__ __ <>__
/ /_/ /__ __/ /
/ __ / _ `/ / /
/_/ /_/\_,_/_/_/ version 0.2.62-84fa81b9ea3d
LOGGING: writing to /home/jupyter/hail-20211203-1940-0.2.62-84fa81b9ea3d.log
2021-12-03 19:41:03 Hail: INFO: Reading table without type imputation
Loading field 'f0' as type str (user-supplied)
Loading field 'f1' as type int32 (user-supplied)
Loading field 'f2' as type int32 (user-supplied)
2021-12-03 19:41:03 Hail: INFO: Reading table without type imputation
Loading field 'SampleID' as type str (not specified)
Loading field 'FamID' as type str (not specified)
Loading field 'Role' as type str (not specified)
2021-12-03 19:41:23 Hail: INFO: Coerced sorted dataset
2021-12-03 19:41:26 Hail: WARN: entries(): Resulting entries table is sorted by '(row_key, col_key)'.
To preserve row-major matrix table order, first unkey columns with 'key_cols_by()'
2021-12-03 19:41:26 Hail: INFO: Reading table without type imputation
Loading field 'FamID' as type str (not specified)
Loading field 'TrioID' as type str (not specified)
Loading field 'SampleID' as type str (not specified)
Loading field 'FatherID' as type str (not specified)
Loading field 'MotherID' as type str (not specified)
2021-12-03 19:41:27 Hail: WARN: export_vcf: ignored the following fields:
'pheno' (column)
'num_alleles' (row)
'a_index' (row)
'was_split' (row)
2021-12-03 19:41:43 Hail: INFO: Coerced sorted dataset
2021-12-03 19:41:55 Hail: INFO: Coerced sorted dataset
2021-12-03 19:42:32 Hail: INFO: Coerced sorted dataset
2021-12-03 19:42:41 Hail: INFO: Coerced sorted dataset
2021-12-03 19:42:51 Hail: INFO: Coerced sorted dataset
2021-12-03 19:43:15 Hail: INFO: Coerced sorted dataset
2021-12-03 19:43:28 Hail: INFO: Coerced sorted dataset
2021-12-03 19:43:37 Hail: INFO: Coerced sorted dataset
2021-12-03 19:44:00 Hail: INFO: Coerced sorted dataset
2021-12-03 19:44:27 Hail: INFO: Ordering unsorted dataset with network shuffle
2021-12-03 19:44:27 Hail: INFO: Ordering unsorted dataset with network shuffle
2021-12-03 19:44:44 Hail: INFO: Ordering unsorted dataset with network shuffle
2021-12-03 19:45:55 Hail: INFO: merging 1 files totalling 45.8K...
2021-12-03 19:45:55 Hail: INFO: while writing:
mssng_db6.chr12_filter_ME_v3_loosefilters.vcf.bgz
merge time: 131.775ms
Processing:12

jupyter@saturn-282f6108-da43-4df8-95ea-903fb699127b-m:~$ ls
hail-20211203-1859-0.2.62-84fa81b9ea3d.log notebooks
hail-20211203-1912-0.2.62-84fa81b9ea3d.log stdout.txt
hail-20211203-1931-0.2.62-84fa81b9ea3d.log wgs_pre_processing_vcf_v3_Loose_Filters.py
hail-20211203-1940-0.2.62-84fa81b9ea3d.log

And if I give it a gs:// uri or a complete path I get a java error:

[Stage 13:> (0 + 1) / 1]2021-12-02 15:14:39 Hail: INFO: Coerced sorted dataset
[Stage 14:> (0 + 1) / 1]2021-12-02 15:14:47 Hail: INFO: Coerced sorted dataset
[Stage 16:> (0 + 1) / 1]2021-12-02 15:15:12 Hail: INFO: Coerced sorted dataset
[Stage 18:> (0 + 1) / 1]2021-12-02 15:15:39 Hail: INFO: Ordering unsorted dataset with network shuffle
2021-12-02 15:15:39 Hail: INFO: Ordering unsorted dataset with network shuffle
[Stage 21:> (0 + 1) / 1]2021-12-02 15:15:55 Hail: INFO: Ordering unsorted dataset with network shuffle
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1212, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1038, in send_command
response = connection.send_command(command)
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1217, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File "wgs_pre_processing_vcf_v3_Loose_Filters.py", line 124, in <module>
trim_vcf(vcf_uri, lcr_uri, ped_uri, meta_uri, trio_uri, vcf_out_uri, 'GRCh38')
File "wgs_pre_processing_vcf_v3_Loose_Filters.py", line 109, in trim_vcf
hl.export_vcf(mt, vcf_out_uri)
File "<decorator-gen-1393>", line 2, in export_vcf
File "/opt/conda/lib/python3.7/site-packages/hail/typecheck/check.py", line 577, in wrapper
return __original_func(*args_, **kwargs_)
File "/opt/conda/lib/python3.7/site-packages/hail/methods/impex.py", line 538, in export_vcf
Env.backend().execute(ir.MatrixWrite(dataset._mir, writer))
File "/opt/conda/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 74, in execute
result = json.loads(self._jhc.backend().executeJSON(jir))
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1310, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/opt/conda/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 16, in deco
return f(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o122.executeJSON
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35705)
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 982, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1120, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35705)
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 982, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1120, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35705)
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 982, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1120, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35705)
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 982, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1120, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35705)
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 982, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1120, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35705)
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 982, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1120, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused

Comments

0 comments

Please sign in to leave a comment.